Critical Flaw in Replicate AI Platform Exposes Proprietary Data

A critical vulnerability in the Replicate AI platform could have allowed attackers to execute a malicious AI model within the platform for a cross-tenant attack — allowing access to the private AI models of customers and potentially exposing proprietary knowledge or sensitive data.

Researchers at Wiz discovered the flaw as part of a series of partnerships with AI-as-a-service providers to investigate the security of their platforms. The discovery of the flaw demonstrates the difficulty of tenant separation across AI-as-a-service solutions, especially in environments that run AI models from untrusted sources.

"Exploitation of this vulnerability would have allowed unauthorized access to the AI prompts and results of all Replicate's platform customers," and potentially alter those results, Wiz's Shir Tamari and Sagi Tzadik wrote in a blog post published today. Previously, Wiz researchers found flaws that led to a similar outcome in the HuggingFace AI platform.

"As we saw in the results of our work with Hugging Face and now in Replicate, two leading AI-as-a-service providers, when running AI models in cloud environments, it is crucial to remember that AI models are actually code," Ami Luttwak, Wiz CTO and co-founder, tells Dark Reading. "Like all code, the origin must be verified, and content-scanned for malicious payloads."

Indeed, the flaw presents an immediate threat to AI-as-a-service providers, who often allow their customers to execute untrusted code in the form of AI models in shared environments - where there is other customers' data. It also can impact AI teams, who could be affected when they adopt AI models from untrusted sources and run them on their workstation or company servers, the researchers noted.

Wiz Research responsibly disclosed the vulnerability to AI model-sharing vendor Replicate in January 2023; the company promptly mitigated the flaw so that no customer data was compromised. At this time, no further action is required by customers.

Exploiting the Flaw

The flaw lies in achieving remote code execution on Replicate's platform by creating a malicious container in the Cog format, which is a proprietary format used to containerize models on Replicate. After containerizing a model using Cog, users can upload the resulting image to Replicate's platform and start interacting with it.

Wiz researchers created a malicious Cog container and uploaded it to the platform and then, with root privileges, used it to execute code on the Replicate infrastructure.  

"We suspect this code-execution technique is a pattern, where companies and organizations run AI models from untrusted sources, even though these models are code that could potentially be malicious," the researchers wrote in the post. A similar technique was used to exploit flaws found on the HuggingFace platform.

This exploitation allowed the researchers to investigate the environment move laterally out and ultimately outside of the node on which they were running, which was inside a Kubernetes cluster hosted on Google Cloud Platform. Though the process was challenging, they eventually were able to conduct a cross-tenant attack that allowed them to query other models and even modify the output of those models.

"The exploitation of this vulnerability would have posed significant risks to both the Replicate platform and its users," the researchers wrote. "An attacker could have queried the private AI models of customers, potentially exposing proprietary knowledge or sensitive data involved in the model training process. Additionally, intercepting prompts could have exposed sensitive data, including personally identifiable information (PII)."

Indeed, this ability to alter prompts and responses of an AI model poses a severe threat to the functionality of AI applications, giving attackers a way to manipulate AI behavior and compromise the decision-making processes of these models.

"Such actions directly threaten the accuracy and reliability of AI-driven outputs, undermining the integrity of automated decisions and potentially having far-reaching consequences for users dependent on the compromised models," the researchers wrote.

New Forms of Mitigation Required

Currently there is no easy way to validate a model's authenticity, or to scan it for threats, so malicious AI models present a new attack surface for defenders that needs other forms of mitigation, Luttwak says.

The best way to do this is to ensure that production workloads only use AI models in secure formats, like so-called safetensors. "We recommend that security teams monitor for usage of unsafe models and work with their AI teams to transition into safetensors or similar formats," he says.

Using only safe AI formats can the attack surface "dramatically," as "these formats are designed to prevent attackers from taking over the AI model instance," Luttwak says.  

Further, cloud providers who run their customers' models in a shared environment should enforce tenant-isolation practices to ensure that a potential attacker who managed to execute a malicious model cannot access the data of other customers or the service itself, he adds.