The iLO5 chipset provides an unprecedented level of hardware security with its silicon root of trust. The silicon root of trust:
- Is based in the silicon chip hardware itself
- Is virtually impossible to alter
- Enables firmware to be authenticated as far back as the supply chain
- Provides a secure startup process
I've spent a lot of time trying to RE the firmware for these devices but never got past the encryption. Someone obviously has, very interesting to see what comes out in this space in future.
SecureROM in the iPhone 6 didn't properly configure the MMU before initialising the NAND, so it turned out to be possible to interpose between the NVME controller and main CPU (which had just started using PCI-e fabric for I/O, IIUC), take advantage of a bunch of software pieces that happened to be sitting in all the right places, and have fun with DMA bus mastering :3
It's like consoles, you can spend many many hours setting up your ultra-secure code signing boot procedure but what good is any of that if you use it to sign firmware with an ancient WebKit version that allows code execution?
Same thing that happened here, the code signing works fine but it was used to sign code with trivial format vulnerabilities running in an environment (RTOS) that doesn't support any sort of isolation.
(FWIW: The Blackhat link elsewhere in the thread clarifies that the root-of-trust feature is new in Gen10/ILO5. I don't know a good source to cite, but it sounds plausible that ILO<=4 might not use 202x-vintage secure boot. Not that that changes much, given that the talk is about breaking the latest generation...)
> The iLO5 chipset provides an unprecedented level of hardware security with its silicon root of trust. The silicon root of trust: - Is based in the silicon chip hardware itself - Is virtually impossible to alter
This is pretty much standard today (and has been for some years). See ARM security model and Trusted Firmware design documents.
The strangest part here is that the malware sample they obtained does nothing but periodically wipe the server disks. That seems like it would be relatively low impact (people have backups, right?) and a surefire way to get found out.
I suspect this isn't the original implant, but rather a 'clean up' one. The APT probably had already obtained all the intelligence they want; now their goal is to make the server appear to misbehave, so it gets taken out and recycled for e-waste.
The Wikipedia article you linked does not mention the motivation for the attack, but your comment seems to say that the motivation was to make others aware of this type of vulnerability. If you have any other links which talk about that, please include one of them.
I am not at all shocked by this. FWIW, iLOs have unique highly randomized default credentials, but I'd guess the vast majority are never even touched, much less thought about, by the people who manage those systems.
>but I'd guess the vast majority are never even touched, much less thought about, by the people who manage those systems.
Really? I'm certainly no big data center sheep dog, but even for the tiny handful of systems I manage IPMI/BMC always seemed to come with big, flashing red lights just from their mere description. They're enormously powerful and valuable tools, but seems pretty obvious that can work both ways. Beyond their own credentials (and it's a real shame so few support smartcards/hardware tokens for login), having them on their own isolated VLAN seemed about the bare minimum. If budget allows/there are enough systems flat out having their own 100% independent physical infrastructure isn't such a stretch either, not like a simple switch and console system or if necessary a decent minimal bastion is that big a price tag these days. In some cases may make the most sense to just use them to setup then disconnect them.
You may well be right but still, yeesh. It's remote super root access, should make any sysadmin treat it with healthy fear from just a description. Although what did give me the willies was how it seems common to have IPMI be hybrid with LAN1 by default rather then it's own physically independent port. I switched from HPE to SuperMicro kit because at our scale and application it made sense for a lot of reasons anyway, but that HPE wants a lot of extra money for a dedicated port and full fat features was a real factor.
Unfortunately I have to concur with GP, my experience in penetration testing is that many sysadmins seem oblivious to the existence of ILO systems, or don't believe they can cause any harm. Alternatively, sometimes one simply slips through the cracks, and all it takes is one system with cached credentials to wreck havoc to a network.
Ironically, the more expensive ILO systems can be easier to exploit. Just throw a Linux live rescue image onto virtual media, reboot the machine, and you can grab everything sensitive fairly quickly.
Well (and to sibling comment), TIL, thank you :(. That's scary though, that anyone could even use the BMC and not at least pause a bit on "wait a sec, what could someone else do with this", same as hypervisor access.
Though maybe some of the blame really should be on the vendors themselves here. At least credentials are randomized now, but they could be much more aggressive about making it something that has to be actively turned on, always has it's own physically isolated port, and requires some level of secure setup (won't communicate untagged maybe?). Or these days making hardware token/smartcard usage required perhaps. There's clearly tradeoffs to be made in terms of security vs convenience/recovery from bad setups, and the balance has fallen fairly decisively in favor of the latter. Kind of depressing though the industry constantly has to do the same thing of ignoring security right up until it gets really, really bad.
In their view, they place a great deal of importance on security; at least for marketing purposes. Sustaining a secure system after it's sold and work has begun on marketing the next new supersecure thing... that's a different story
They were either not plugged in, or definitely not on their own vlan in any of the datacenters (dozens of them) I worked in from 2000-2007. Many admins have no clue they exist or what they do.
On most boards where the ILOM / BMC has a dedicated LAN port it usually also has access to at least one of the main ethernet ports on the MAC level to support hooking machines up with just one cable without loosing ILOM.
> Just isolating on a separate network isn’t enough — if an attacker compromises a single machine, then they get access to that network.
Eh, no. You'd isolate it on a VLAN where only people who need access to ILO have access to. Now, lets zoom out for a moment, does the sales dept. need access to ILO? No. Does the CEO need access to ILO? No. So machines on these VLANs do not have access to that VLAN. And, if the machine of a sysadmin is compromised, you're into deep shit as it is.
Suppose you have 500 servers. You carefully put each of the 500 iLOs on the special iLO VLAN, you can enable 802.1x or MACSET or magical locked Ethernet jacks or whatever and make absolutely certain that only perfectly trustworthy IT admins using perfectly trustworthy computers can touch that network.
And you still lose! Because an attacker can compromise an unimportant sales computer, escalate to root (or SYSTEM), and compromise that computer’s iLO via the internal pretend-PCI transport as discussed in the OP. And now the attacker is on the supposedly secure iLO VLAN.
Replace iLO with any other BMC or BMC-like solution (AMT, for example), and the scope of this issue should be apparent.
To mitigate this, either proper hardware rooted security for BMCs is needed (giving a strong zero-trust model) or a very carefully configured network that isolates all hosts from each other. Or, preferably, both.
It could be an unimportant sales server. [0] Not everyone has a fleet of essentially interchangeable Kubernetes servers, and a credible threat model should allow compromise of a single server without permanently infecting the entire fleet with an APT.
[0] Windows Server, Citrix, etc are real. Just because it has an old fashioned GUI doesn’t mean it doesn’t have rack ears and a management port.
I think the sad truth is that the very long swiss cheese record of this stuff just fits right into, and contributes to, the "secure internal network" mental model of most on-premises cultures. It also benefits the vendors, because they can sell more feebly secured on-prem stuff.
Most iLOMs are some small/cut-down Linux distribution. Sadly often woefully out of date to begin with (both in terms of kernel and userspace apps) and often prematurely abandoned in terms of firmware updates from the OEM. Of course the hardware is just novel enough to make rolling your own install/updates from a standard distribution impossible.
Green Hills Integrity has a pretty good reputation in the embedded space. I imagine this (and other iLO vulnerabilities) are in HPs own firmware modules and interfaces and not in the OS itself.
Quote:
The iLO5 chipset provides an unprecedented level of hardware security with its silicon root of trust. The silicon root of trust: - Is based in the silicon chip hardware itself - Is virtually impossible to alter - Enables firmware to be authenticated as far back as the supply chain - Provides a secure startup process
I've spent a lot of time trying to RE the firmware for these devices but never got past the encryption. Someone obviously has, very interesting to see what comes out in this space in future.