'Cloudborne': Bare-Metal Cloud Servers Vulnerable to Attack

Firmware vulnerabilities in so-called bare-metal cloud servers let attackers install malware and backdoors, which remain active and grant access as servers are reassigned to new customers.

Researchers at Eclypsium are today releasing a report on firmware security issues they believe represent "a fundamental gap" in cloud infrastructure security. Their findings show baseboard management controllers (BMC) built into cloud servers could put customers at risk. While their study is based on IBM SoftLayer technology, they emphasize other providers may be exposed.

"This is a huge industry issue," says Yuriy Bulygin, Eclypsium founder and CEO, who formerly led the advanced threat research team at Intel Security.

With most infrastructure-as-a-service (IaaS) offerings, customers share resources on a physical server. Some organizations, however, have high performance requirements for certain applications or sensitive information they don't want on a machine shared with other firms.

In these cases, providers offer bare-metal cloud services. Customers buy full access to a dedicated physical server they can use however they want, without worrying it will interfere with others' data or buying and supporting additional hardware. When they're done using a bare-metal server, it's reclaimed by the provider, wiped, and repurposed for future customers.

Bare-metal cloud provides certain advantages; for example, performance improvement and the ability for businesses to install their own software stack. It also introduces new security risks as attackers have direct hardware access. This isn't the first time Eclypsium has published findings on firmware flaws: last June, they published findings on vulnerabilities in Supermicro systems.

What is Cloudborne?

Now, researchers say, bare-metal servers may not be fully erased before future use. The vulnerability, which they dubbed Cloudborne, is in the BMC – a privileged component used to manage the server. Using the Intelligence Platform Management Interface (IPMI), admins can send commands to the server or modify/reinstall an OS without physical access to the machine.

Vulnerabilities in the BMC could allow any customer to leave a backdoor on the server. "It's a fundamental gap in the cloud infrastructure, and it's exaggerated in bare-metal cloud infrastructure," says Bulygin. "The problem is that a customer – potentially a malicious customer – of a cloud service provider can have access to bare-metal instances," on which they can modify firmware and infect future users of the same machine with data theft, ransomware, and other threats.

Eclypsium conducted an experiment using IBM's SoftLayer cloud server platform, which offers bare-metal instances in most of its 35 global data centers. The team initially chose SoftLayer because of its simplified logistics and hardware access, as they explain in a blog post. But researchers also noticed Softlayer used Supermicro hardware, which based on earlier research they knew as vulnerable.

Researchers bought access to a bare-metal server, verified it was running the latest BMC firmware, and noted the product chassis and serial numbers for future identification. They made a minor change – a single bitflip inside a text comment they had prepared – and created an additional IPMI user, which they gave administrative access to the BMC channels.

They returned the server to IBM, which conducted the reclamation process, and were later able to reacquire the same server. While the new IPMI account was gone, their change to the BMC firmware remained. Researchers say this shows the BMC firmware wasn't re-flashed during reclamation, which they say makes it possible to implant malicious code into the firmware and steal data from future users.

Researchers also noticed the BMC logs were retained across provisioning, as was the root password. Since the logs were not deleted, future customers could view the actions of previous server owners and attackers could use the root password for future access.

"Most people aren't doing any verification," says John Loucaides, vice president of engineering at Eclypsium, of the reclamation process. "Most people ignore the whole firmware layer altogether." Given IBM is a large player and was affected by this issue, he anticipates other companies in the industry are affected as well.

BMC Bugs Have Been Found Before

This isn't the first time security experts found evidence of Supermicro BMC issues affecting bare-metal cloud servers. It has been a few years since researchers at Rapid7 found security issues in the Supermicro IPMI firmware, used in the BMC of Supermicro motherboards. At the time, HD Moore, then its chief research officer, analyzed the issue related to bare-metal cloud servers. Rapid7's results were similar to Eclypsium's, he says, but at the time the team felt publicly disclosing an insecure process from a specific provider wouldn't benefit the public.

"That equation has shifted a bit with consolidation among providers and the much broader adoption of cloud services," Moore says. Now, he says, Eclypsium's research is "an important problem" and "something both customers and providers should be aware of."

A compromised Supermicro BMC can be used to attack the host operating system in several says, he continues. The most straightforward is via the built-in kernel-based virtual machine (KVM) and remote media boot functionality. An attacker who installs a backdoor into a cloud server can use their access to assume control of the operating system and read the affected customer's hard drive data.

However, mitigating the problem is tough. An attacker with server access can bypass authentication when using IPMI over keyboard controller style (KCS), and create administrative accounts or flash a malicious image to the BMC, as Eclypsium did. Reflashing is handled by BMC firmware, so attackers have access even if the provider restores to a factory version.

IBM's Response

Eclypsium notified IBM of their findings; in response, IBM published a blog post indicating it has addressed the issue, and there is no evidence it has been exploited for malicious purposes.

IBM reports it is forcing all BMCs, including those reporting up-to-date firmware, to be reflashed with factory firmware before they are re-provisioned for future customers. It erases all logs in BMC firmware and regenerates all passwords for the firmware, officials report.

"IBM's approach to sanitizing servers before redeploying them is a good start, but not a complete resolution," says Moore. The firmware update process can be compromised with malicious firmware; an attacker that flashes a custom firmware can prevent providers from possibly detecting the backdoored image. He also notes that public tools exist to create custom firmware images for Supermicro components; attackers can use these to achieve access.

Researchers take issue with the fact that IBM categorized this issue as "low severity." Using the CVSS 3.0, they classified the problem as 9.3, or critical severity. "It's not a low-severity issue by any means," Loucaides says.

Related Content:

Join Dark Reading LIVE for two cybersecurity summits at Interop 2019. Learn from the industry's most knowledgeable IT security experts. Check out the Interop agenda here.