Cloudflare Bleeds Bad News – & Good

"For the want of a nail the shoe was lost, For the want of a shoe the horse was lost, For the want of a horse the rider was lost, For the want of a rider the battle was lost, For the want of a battle the kingdom was lost, And all for the want of a horseshoe-nail."
— Benjamin Franklin

Substitute the ">" symbol for an old-fashioned nail, and you have a synopsis of the latest privacy catastrophe, dubbed "Cloudbleed" by security researchers. How bad is Cloudbleed? The advice to users echoed around the Internet gives a hint of the breach's scope:

Change your passwords. All of them.

What happened to cause the problem and how it has been addressed each carry instructions for enterprise security and IT professionals. What happened is simple, and the sort of coding error that can be frustratingly difficult to catch absent the most rigorous testing protocols. In a single line of code, the intended ">=" operator was typed "==". As a result, certain operations were able to fill a buffer and keep right on writing, planting data across the system.

The flaw
According to Tavis Ormandy, a researcher with Google Project Zero, that planted data can include cookies, private messages, IP addresses, and passwords. The worst part? In a blog post about the event (made public only after the vulnerability had been repaired and remediated), Ormandy wrote, "PII [personally identifiable information] was actively being downloaded by crawlers and users during normal usage, they just didn't understand what they were seeing."

Cloudflare is a service provider with a client list that includes Uber, OKCupid, Fitbit, and various financial institutions. In addition to basic network performance services that include DNS and load balancing, Cloudflare provides security capabilities such as SSL and DDoS mitigation. Because Cloudflare provides the service to businesses rather than consumers, it's nearly impossible for an individual to know whether their personal information was exposed. And it's absolutely impossible to know whether any exposed information was actually grabbed by criminals -- until, of course, some form of identity theft takes place. Hence, the advice to change all your passwords. Now.

The problem was compounded by the way that Cloudflare operates. Cloudflare's data can be crawled and cached by services such as Google. This means that personal data was not just exposed, it was exposed in a persistent manner. That's a lot of bad news wrapped up in a single wayward character. Fortunately, there is some good news in the story to balance a bit of the bad.

Rapid response
When Ormandy discovered the issue, he immediately contacted Cloudflare using Twitter (in, as one commenter noted, the sort of message that should be absolutely horrifying to anyone on a corporate security team). In very short order, Cloudflare security personnel had reproduced the issue and shut down the services making use of the affected code. According to the company, they then put teams in the US and UK on 12-hour shifts, handing code off between the teams to keep efforts to find and remediate the problem going 24 hours a day.

Cloudflare reports that all services that included the problem were disabled globally within roughly three hours of notification. Figuring out precisely where in the code the problem lay, and how it could be repaired, took a bit longer. The company's blog post on the issue (referenced above) goes into great detail about where the problem was, what its effects were, and why it had not previously appeared. The level of detail in the disclosure is part of the good news surrounding the vulnerability; in speaking with several security researchers and IT professionals, all said that this disclosure is a model of what they would like to see in future industry events. All felt that this level of disclosure is likely to assuage customer fears and reassure regulators that all proper steps have been taken.

The final step in remediation was Cloudflare working with search engines to clear caches of released information. As the company wrote in its disclosure blog post, "With the help of Google, Yahoo, Bing and others, we found 770 unique URIs that had been cached and which contained leaked memory. Those 770 unique URIs covered 161 unique domains. The leaked memory has been purged with the help of the search engines."

Six days after the initial contact from Ormandy, Cloudflare disclosed the incident to the public. The vulnerability had existed for approximately four months and the number of customer records revealed to criminals is unknown. What is known is that this will likely be used as an example of the white-hat research system working as intended; independent discovery of an issue, verification by the affected company, rapid repair and remediation, with speedy, transparent disclosure to the public.

— Curtis Franklin, Security Editor, Light Reading