Accidental Heartbleed Vulnerabilities Undercut Recovery Effort

Update (5/13/2014): After working with F5 to track down the apparent surge in machines sporting the Heartbleed vulnerability, researcher Yngve Petterson has revised his findings: "Due to an issue with the network connection of the prober the test used to detect F5 BigIP [servers, it] showed higher numbers than it should have, and the numbers of such servers therefore got very inflated for the scans that were run in the past month. This means that the BigIP-related information and conclusions are not correct. ... My apologies to F5 and their customers for this mistake."

10 Ways To Fight Digital Theft & Fraud

Heartbleed recovery efforts are being partially undermined by new servers coming online that use a vulnerable version of the popular OpenSSL cryptographic software library.

That warning was sounded by software developer Yngve Pettersen at Vivaldi Technologies, based on ongoing scans he's conducted -- focusing on sites listed on the Alexa list of the world's million most popular websites that sport the OpenSSL vulnerability. He began regularly scanning about 500,000 different servers, 20% of which include Heartbeat TLS Extension support, after the vulnerability was discovered in early April.

The good news is that overall rates have sharply declined. "In the six scans I have made since April 11, the number of vulnerable servers [has] trended sharply downward, from 5.36% of all servers, to 2.33% this week," Pettersen said Wednesday in a blog post. In fact, he said 75% of all vulnerable servers appeared to have been patched within four days of the bug being discovered.

Other scanning efforts have also charted a steep decline in the number of vulnerable servers. For example, port 443 scans conducted by Errata CEO Robert Graham one month ago found that more than half a million servers appeared to have the vulnerability. Since then, however, the number of vulnerable systems appears to have halved.

[Here's what you should know about recent authentication-protocol implementation security flaws: OAuth, OpenID Flaw: 7 Facts.]

"Whereas my previous scan a month ago found 600,000 vulnerable systems, today's scan found roughly 300,000 thousand systems -- 318,239 to be precise," Graham said Thursday in a blog post. That was out of a total of 1.5 million systems he detected that support the "heartbeat" feature. "Note that only OpenSSL supports heartbeats, meaning that the vast majority of SSL-supporting servers are based on software other than OpenSSL," he said. Graham also emphasized that he scanned IPv4 addresses, which won't produce like-for-like comparisons with other types of scans, for example of the DNS domain names Pettersen scanned.

While the overall drop in vulnerable sites -- by either measure -- is good news, based on his ongoing scans, Pettersen has unearthed two problems. First, many patched servers are still using their old digital certificates. "Given that any server that was patched after April 7 has to be assumed to have had its certificate private key compromised -- because criminals may have used Heartbleed to compromise their server -- this indicates a serious problem for the users of those sites," Pettersen said.

{image 1}

Second, there's been an alarming rise in the number of new servers that sport Heartbleed, including a sizeable number of F5's BigIP crypto accelerator servers. "In my most recent scan, 20% of the currently vulnerable servers -- as distinguished by IP addresses -- and 32% of the vulnerable BigIP servers were NOT vulnerable when they were scanned previously," he said. "This means that thousands of sites have gone from not having a Heartbleed problem to having a Heartbleed problem."

Pettersen sees two likely explanations for the increase in new servers harboring Heartbleed. First, the number of BigIP servers he's spotting has doubled in the past month, likely because F5 customers have brought new BigIP servers online. Evidently, many of those servers are using older, vulnerable firmware and haven't yet been patched to eliminate Heartbleed.

Another potential explanation for the surge in newly vulnerable machines is that some IT administrators may have thought that their OpenSSL-using servers were vulnerable to Heartbleed when they weren't. "This, perhaps combined with administrative pressure and a need to 'do something,' led them to upgrade an unaffected server to a newer but still buggy version of the system, perhaps because the system variant had not yet been officially patched," Pettersen said.

Going forward, patch managers will need to keep a close eye on version control to avoid accidentally upgrading to a Heartbleed-vulnerable version of their enterprise software before a patched version becomes available. "We expect to see administrators update to a fixed version, revoking and re-issuing certificates, and take various other steps that are applicable for their setup, as soon as possible," said Kasper Lindgaard, director of research and security at Secunia, via email.

But patches are not yet available for all vulnerable products. Accordingly, Lindgaard said, "We are not yet at a stage where everybody can simply patch. Some vendors are still in the process of identifying vulnerable products and developing patches. It will definitely take some time before we reach the milestone where it is only the end users that need to patch."

Cyber-criminals wielding APTs have plenty of innovative techniques to evade network and endpoint defenses. It's scary stuff, and ignorance is definitely not bliss. How to fight back? Think security that's distributed, stratified, and adaptive. Read our Advanced Attacks Demand New Defenses report today. (Free registration required.)