IBM Cloud Supply Chain Vulnerability Showcases New Threat Class

A vulnerability in IBM Cloud databases for PostgreSQL could have allowed attackers to launch a supply chain attack on cloud customers by breaching internal IBM Cloud services and disrupting the hosted system's internal image-building process.

Security researchers from Wiz discovered the flaw, which they dubbed "Hell's Keychain." It included a chain of three exposed secrets paired with overly permissive network access to internal build servers, the researchers revealed in a blog post published Dec. 1. 

While now patched, the vulnerability is significant in that it represents a rare supply-chain attack vector impacting the infrastructure of a cloud service provider (CSP), Wiz CTO Ami Luttwak tells Dark Reading. The discovery also uncovers a class of PostgreSQL vulnerabilities affecting most cloud vendors, including Microsoft Azure and Google Cloud Platform.

"This is a first-of-a-kind supply-chain attack vector, showing how attackers might be able to leverage mistakes in the build process to take over the entire cloud environment," he says.

Specifically, researchers uncovered "major risk caused by improper sanitation of build secrets from container images, allowing for an attacker to gain write access to the central container image repository," Luttwak says. This would have allowed the actor to run malicious code in customers’ environments and modify the data stored in the database.

"Modifications to the PostgreSQL engine effectively introduced new vulnerabilities to the service," the researchers wrote in their post. "These vulnerabilities could have been exploited by a malicious actor as part of an extensive exploit chain culminating in a supply-chain attack on the platform."

As mentioned, the ability to use PostgreSQL to breach IBM Cloud is not unique to the service provider, researchers said. Wiz already has found similar vulnerabilities in other CSP environments, which they plan to disclose soon and which highlight a broader issue of cloud misconfigurations that pose a supply chain threat to enterprise customers.

The existence of the flaw also highlights how improper management of secrets — or long-lived authentication tokens for cloud APIs or other enterprise systems — can impose a high risk of unwanted intrusion by attackers on an organization using a cloud provider, Luttwak says.

"Finding and utilizing exposed secrets is the No. 1 method for lateral movement in cloud environments," he says.

For now, the researchers said they worked with IBM to remedy the issue in IBM Cloud and no customer mitigation action is required.

Uncovering the Chain

Researchers were doing a typical audit of IBM Cloud's PostgreSQL-as-a-service to find out if they could escalate privileges to become a "superuser," which would allow them to execute arbitrary code on the underlying virtual machine and continue challenging internal security boundaries from there.

Based on their experience, they said the ability to carry out a supply chain attack on a CSP lies in two key factors: the forbidden link and the keychain.

"The forbidden link represents network access — specifically, it is the link between a production environment and its build environment," the researchers wrote. "The keychain, on the other hand, symbolizes the collection of one or more scattered secrets the attacker finds throughout the target environment."

On its own, either scenario is "unhygienic," but not critically dangerous. However, "they form a fatal compound when combined," the researchers said.

Hell’s Keychain held three specific secrets: a Kubernetes service account token, a private container registry password, and continuous integration and delivery (CI/CD) server credentials.

Combining this chain with the so-called forbidden link between Wiz's personal PostgreSQL instance and IBM Cloud databases’ build environment allowed researchers to enter IBM Cloud’s internal build servers and manipulate their artifacts, the researchers said.

Implications for Cloud Security

The scenario presented in Hell's Keychain represents a broader problem within the cloud security community that demands attention and remediation, the researchers said. To wit: scattered plaintext credentials that are found across cloud environments that impose a huge risk on an organization, impairing service integrity and tenant isolation, they said.

For this reason, secret scanning at all stages of the pipeline is crucial, including in CI/CD, code repo, container registries, and within the cloud, Luttwak says.

"Furthermore, lockdown of privileged credentials to the container registry is crucial, as these credentials are often overlooked but are actually the keys to the kingdom," he adds.

CSP customers also should consider image signing verification via admission controllers to ensure these sort of attacks are prevented entirely, Luttwak says.

Hell's Keychain also highlights a common misconfiguration in the use of the popular Kubernetes API for container management within the cloud — pod access, ''which can lead to unrestricted container registry exposure," he says.

Another best practice the researchers recommend is any organization — CSP or otherwise — deploying a cloud environment can take is to impose strict network controls between the Internet-facing environment and the organization's internal network in the production environment, so attackers can't gain a deeper foothold and maintain persistence if they do manage to breach it.