Google Kubernetes Clusters Suffer Widespread Exposure to External Attackers

The authentication mechanism within the Google Kubernetes Engine (GKE) has a loophole that could allow an external attacker with any Google account to access organizations' private Kubernetes container clusters, researchers have found.

This could lead to serious cloud security incidents, such as cryptomining, denial-of-service (DoS), and the theft of sensitive data, Orca Security warned.

The Orca Research Pod discovered the issue, dubbed Sys:All, which arises when users grant Kubernetes privileges to the "system:authenticated" group, which includes all users with a Google account. Admins may grant access to this Google-curated group — or "bind" it, in GKE parlance — because they believe it includes only organization-authorized and verified GKE users. In actuality, the group includes any Google authenticated account, including those outside the organization, according to a blog post published Jan. 24.

"This misunderstanding creates a significant security loophole when administrators unknowingly bind this group with overly permissive roles," Roi Nimisi, senior security researcher with Orca, wrote in the post. "Administrators may mistakenly believe this is safe —especially since this would be the case for similar Kubernetes groups in AWS and Azure."

Orca researchers wrote a short Python script scanning for GKE clusters in a known classless inter-domain routing (CIDR) and "with minimal effort" found 250,000 active GKE clusters in the wild that are reachable; of those, 1,300 clusters were potentially vulnerable to Sys:All attacks, and 108 of them could lead to an immediate compromise, allowing either cluster-admin access, clusterwide listing of secrets or clusterwide write/delete actions, the researchers found. Others allow read permissions over different native resources, and read/all permissions over custom resources.

Given that these numbers are only a fraction of the threat landscape, "I would carefully say we can expect over 1 million vulnerable GKE clusters to this attack vector," Nimisi wrote.

Exploiting Security Loopholes in Vulnerable GKE Clusters

GKE is Google's out-of-the-box, managed implementation of the open source Kubernetes container-management tool that includes custom features and integrations, including a seamless OIDC-based authentication with Google as the identity provider.

Kubernetes has emerged as one of the most widely used open source systems for containers; however, it also has become a prime target for threat actors due to its vast potential for exploitation and access to organizational data.

Further, lax permissions have created issues for Kubernetes before. In one such case, an attacker turned a cryptojacking attack into a wide-ranging intrusion that targeted intellectual property and sensitive data by exploiting lax permissions on a Kubernetes cluster running on the Amazon Web Services (AWS) cloud.

As a proof-of-concept, the researchers themselves already managed to exploit the Sys:All loophole to penetrate vulnerable GKE clusters, including those of a Nasdaq-listed company, they revealed in a companion blog post published Jan. 24.

"A seemingly innocuous misconfiguration in the system:authenticated group had far-reaching implications, such as allowing list and pull images from the company’s container registries and providing open access to AWS credentials stored within a cluster’s configmap (alongside other sensitive data found)," Ofir Yakobi, security researcher with Orca, wrote in the post.

The researchers used these credentials to access AWS S3 buckets containing sensitive information and multiple logs that, upon further analysis, revealed system admin credentials and valuable endpoints, including RabbitMQ, Elastic, the authentication server, and other internal systems, all with administrator access.

An exploration of other exposed GKE clusters revealed a variety of sensitive data exposure across multiple organizations, including: exposure of GCP API keys and service account JSONs, discovery of private keys, access to container registries, and access to critical services.

"The cumulative findings from our research painted a concerning picture of the widespread nature of security lapses in cloud environments," Yakobi wrote.

Google's Response & Cyber Risk Mitigations

Orca immediately made Google aware of Sys:All —and the company subsequently released a security bulletin with several prevention measures to address it, even though changing the behavior within the engine "is impractical due to breaking the design," Nimisi wrote.

For instance, Google has blocked the binding of the system:authenticated group to the cluster-admin role in newer versions of the engine (version 1.28 and later). The company also is notifying potentially vulnerable customers and plans further architectural changes in the future.

Despite these mitigations, there are still numerous other roles and permissions other than cluster-admin that can be assigned to the system:authenticated group, so the loophole remains a significant security concern, Nimisi said.

Thus, in addition to upgrading to GKE version 1.28 or higher, organizations with GKE clusters running should strictly follow the principle of least privilege to protect themselves, Orca advises. This means that they should only grant users privileges to cloud assets that they truly need for their specific roles within an organization. These privileges also should be continuously monitored so as not to provide more access than is necessary.

As Google's bulletin noted, "Google's approach to authentication is to make authenticating to Google Cloud and GKE as simple and secure as possible without adding complex configuration steps. Authentication just tells us who the user is; authorization is where access is determined."

Organizations might want to consider using a reputable cloud security platform to help them find all potentially vulnerable Kubernetes clusters across their cloud deployments, tighten permissions, and ensure they are continuously monitored for any future configurations that might occur, Nimisi added.

Google is also actively collaborating with Orca Research to incorporate the findings into its response, according to a Google spokesperson.

“We created our Vulnerability Rewards Program specifically to identify security events with potential customer impact," the person says. "We appreciate Orca Security and the broader security community’s ongoing participation in these programs. We value the work of the researchers and have issued a security bulletin for the limited number of impacted GKE users detailing the steps they should take to protect themselves from any accidental authorization.”