Container Verification Bug Allows Malicious Images to Cloud Up Kubernetes

A high-severity security vulnerability in the Kyverno admission controller for container images could allow malicious actors to import a raft of nefarious code into cloud production environments.

The Kyverno admission controller offers a signature-verification mechanism designed to ensure that only signed, validated container images are being pulled into a given Kubernetes cluster. This can ward off any number of bad outcomes, given that boobytrapped container images can contain payloads as varied as cryptominers, rootkits, exploit kits for container escape and lateral movement, credential stealers, and more.

However, the bug (CVE-2022-47633) can be exploited to subvert that mechanism. "The vulnerability enables an attacker … to inject unsigned images into the protected cluster, bypassing the image verification policy," explained researchers at ARMO, in a blog post on Dec. 21. The stakes are high: The attacker can effectively take control of a victim’s pod and use all of its assets and credentials, including the service account token to access the API server, they warned.

"The vulnerability enables a complete bypass of image signature verification. In the case of a Kubernetes cluster, this gives an attack a wide range of targets. Any workload can mount cluster secrets and data volumes," Ben Hirschberg, CTO and co-founder of ARMO, tells Dark Reading. "This means the attacker can inject code that can steal data and credentials from the Kubernetes cluster of the victim. This also enables the attacker to inject his/her own code and use the CPU of the victim for things like cryptocurrency mining."

Inside the Bug: Subverting the Container Admission Controller

When a new workload, defined via an image with a tag, is requested from a Kubernetes API server, the API server asks the Kyverno admission controller to validate the new workload. To determine whether a workload can be admitted to the cluster, the admission controller requests the image manifest and a signature from the container registry.

If they check out, the image gets the green light, and the container runtime starts a new workload based on said image.

The vulnerability arises because the controller's signature validation process downloads the image manifest twice — but only verifies a signature for one of the downloads, according to the advisory.

Thus, the attack looks like this: An administrator is social-engineered into pulling a container image from a malicious registry or proxy. When the image is first imported, the malicious registry returns a valid, benign, signed image to the admission controller. So far, so good.

However, then the admission controller requests the manifest of the signed image for a second time, to get the digest for mutation — i.e., to update the container's human-readable tag. This time, no signing validation occurs, allowing the malicious registry to return a different, unsigned and malicious image, which is ultimately the one that is spun up and run.

"This is a classic example of a [time-of-check-to-time-of-use] TOCTOU problem that allows the attacker to pull a bait-and-switch," according to ARMO's analysis. "Since the image manifest which will eventually be used is not the same as the one which was verified, this enables the attacker to trick the client."

The vulnerability was introduced in version 1.8.3 and was fixed in version 1.8.5; Kyverno users should update as soon as possible. The patch ensures that the same image hash is used to change the workload specification as was used to verify the signature.

This specific vulnerability affects only Kubernetes with Kyverno, but other image signature verification tools need to take care not to be vulnerable to the same method, Hirschberg warned.

Social Engineering a Malicious Container Attack

To carry out a real-world attack, threat actors can use either compromised accounts on existing registries to host malicious images, or they can establish their own private container registry and then set about convincing an admin to trust it.

From a practical standpoint, "creating a malicious registry for an experienced attacker is not a challenge," Hirschberg says. "An attacker can take any open source registry software, make some minor modifications to make the attack work, and run it in the cloud under a custom domain."

The next step is to convince an admin to trust the malicious container, which is also not that difficult. Container images from third parties are often used to spin up ready-made applications, in much the same way that app developers source prebuilt code blocks from open repositories like npm — the idea is to not have to reinvent the wheel for common functions and utilities.

Hirschberg notes that only a fraction of Kubernetes users have strictures on where they can pull container workloads from, so cloud admins are not likely to be immediately on their guard when it comes to using third-party registries — particularly if they have image signature verification in place.

"The attacker could go phishing and publish in multiple forums a notification that there is a new version of software XYZ, and here are the Kubernetes YAML or Helm to run it," he explains. "Since some people feel protected by image signature verification, their guard would be down and would not be afraid to run the image."

Container Security: A Rising Concern

Containers are a good target for cybercriminals because they mostly run in the cloud with access to plenty of computational resources, which are precious and expensive, Hirschberg points out — so, this enables attackers to steal computational resources and data, while also going unnoticed for a relatively long period of time.

"We don’t have exact statistics, but it is very clear that with the wide adoption of containers, this is becoming a more prevalent issue," he says. "Security teams are learning how to handle them, and Kubernetes in general. I don't think that it is a true 'blind spot,' but container security teams are still learning the whole environment with many neglected areas."

With the adoption of image signature verification still in its early stages, admission controllers represent one of those potentially neglected areas. But they're also part of a broader conversation about supply chain software security, that should be put in the spotlight.

"The SolarWinds attack showed the world how sensitive this issue is when it comes to trusting the security of external code," Hirschberg says. "Kyverno is among the first security tools to implement signature validation, and with new features can come new bugs. Hopefully, this finding makes this a more secure mechanism and will help the industry to overcome the problem of verifying software in Kubernetes."