ChaosDB: Researchers Share Technical Details of Azure Flaw

BLACK HAT EUROPE 2021 — LONDON — Researchers who discovered a severe vulnerability in the Microsoft Azure Cosmos DB database solution today revealed the full extent of the flaws they found and previously undisclosed details of their investigation, which it turns out was far more extensive than first revealed.

In August 2021, the Wiz team revealed a critical vulnerability in the Azure cloud platform that would enable remote account takeover of the Cosmos DB database. Dubbed ChaosDB, this flaw gave any Azure user full admin access to other customers' Cosmos DB instances without authorization. Its impact spanned thousands of businesses, including many Fortune 500 firms.

More specifically, multiple flaws existed in Microsoft's implementation of Jupyter Notebook, an open source Web application commonly used for data science. A local privilege escalation flaw led to unrestricted network access, which allowed researchers to access a wide range of certificates and private keys that provided admin access to other users' Cosmos DB accounts.

To make things worse, Cosmos DB accounts previously came with Jupyter Notebook auto-enabled, which wasn't made clear to users. As a result, many customers were unknowingly exposed to this vulnerability.

Wiz reported the findings to Microsoft, which issued a fix within 48 hours and confirmed in a blog post that no customer data had been accessed using this vulnerability by third parties or security researchers. It also shut down the Jupyter Notebook feature, albeit temporarily.

But this wasn't the full story of Chaos DB, Wiz security researchers Sagi Tzadik and Nir Ohfeld said in their Black Hat talk today. The vulnerability did more than allow an unprivileged user to obtain complete, unrestricted access to databases of several thousand Azure customers.

By exploiting each misconfiguration in Cosmos DB, and chaining them together, the researchers were able to obtain many of Microsoft's internal Cosmos DB-related secrets and credentials. With these, they were able to authenticate as admin to more than 100 Cosmos DB-related management panels in the form of Service Fabric instances, or the container orchestration tool used to power Cosmos DB.

The finding is unprecedented, Ohfeld says in an interview with Dark Reading. "No other person outside of Microsoft gained this kind of administrative access to the magic that actually makes the cloud work." This was one of the reasons, Tzadik adds, that they held off on disclosing their full findings until now — to give the company sufficient time to mitigate the issue. Some of the information they could access was not only about Cosmos DB but about how Azure works.

"Besides taking over the account and manipulating data, we could also have damaged the Cosmos DB service due to the admin position we had from within it," the researchers explained in a blog post. "The impact of gaining access to the underlying Service Fabric instances means that this vulnerability was nearly impossible to defend against as a customer."

Going Down the Rabbit Hole
The team hadn't even been looking for vulnerabilities when their investigation began, Tzadik says. Given the popularity of Cosmos DB, they were initially looking for common misconfigurations and reviewing the solution to spot errors.

While exploring its features, they discovered the embedded Juputer Notebook container, which offers terminal access and the option to interact with the Cosmos DB instance with different programming languages. When they used the Jupyter terminal, or the default Python3 Notebook, they noticed their code was executed as the unprivileged "cosmosuser." When they switched their Python code to C#, they saw the code was being executed with root privileges.

"When we saw the Jupyter Notebook feature, we couldn't resist," says Ohfeld. "As an attacker, when we see a place that lets us execute arbitrary code, we have to have a look."

After discovering the local privilege escalation vulnerability in Jupyter Notebook, they used their root privileges to look around the container to determine which network resources they could access. The researchers found a list of forbidden IP addresses, which they were able to delete as they were configured locally on the container, achieving unrestricted network access.

Their investigation continued from there as Ohfeld and Tzadik continued to explore the previously forbidden IP addresses, discovering access to WireServer, which manages aspects of virtual machines within Azure and the extensions of every Azure VM. They discuss the details in a separate in-depth technical writeup, also published today. Through attempting to uncover secrets and explore the Cosmos DB environment, they were ultimately able to access 25 Microsoft certificates and their corresponding private keys, which Tzadik points to as the moment they knew they were onto something big.

"We were like, OK, this is interesting, let's see what happens," he says.

While the team only used six of the certificates, the ones they used allowed them to obtain the plaintext Primary Key for any Cosmos DB instance running in their cluster, letting them query and manipulate customer databases without authorization. They were able to obtain the plaintext auth token for any Jupyter Notebook instance running in the cluster, as well as plaintext passwords for customers' notebook storage accounts. They could also access the underlying infrastructure of Cosmos DB by accessing internal Azure storage blobs.

"In less than a week of active research, and by using only six of the 25 secrets we obtained, we believe that we were able to nearly take over the entire service," the researchers wrote, noting they gained the same privileges as the internal Microsoft employees who worked on it.

There were many lessons learned here, especially with respect to isolation in the cloud, says Tzadik. "We assumed isolation in the cloud worked properly, and we learned that is not always the case."