11 Heartbleed Facts: Vulnerability Discovery, Mitigation Continue

Just how many products and websites need to be patched, and related digital certificates revoked and reissued, before the Heartbleed vulnerability will be mitigated?

Heartbleed, the recently spotted vulnerability in OpenSSL, could allow attackers to steal websites' private keys. Google engineer Neel Mehta and the Finnish security firm Codenomicon discovered the flaw separately this month. But information about the vulnerability, which later became known as Heartbleed, wasn't made public until OpenSSL issued an April 7 security advisory about a "TLS heartbeat read overrun." At that time, more than half of all web servers, collectively hosting more than 500 million websites, were thought to be vulnerable.

What's the status of Heartbleed vulnerability discovery and related mitigation efforts since then? Here are 11 related facts.

1. Sites: Who patched early?
Before April 7, information about the bug was shared with some organizations -- including Akamai, CloudFlare, and Facebook -- which added safeguards to mitigate the vulnerability, the Sydney Morning Herald reported. Google also informed multiple organizations about the flaw before the information was publicly released, though so far it has declined to name the organizations to which it spoke.

2. Most sites learned about flaw later
However, many other sites appear to have learned about Heartbleed only after OpenSSL issued its April 7 public security advisory. Those sites appear to include Amazon Web Services, Box, Cisco, Dropbox, Flickr, GitHub, GoDaddy, IFTTT, Instagram, Juniper, Netflix, OKCupid, Pinterest, Soundcloud, Tumblr, Twitter, Ubuntu, Vonage, Wikipedia, Wordpress, and Yahoo. Many of those sites have patched the flaw or are in the process of doing so.

3. Good news: Certificate revocations have spiked
What of the millions of other affected sites? Many of them have alrady begun switching out their digital certificates, which is good news. Alex Stanford, research operations manager for the SANS Internet Storm Center, said in a blog post Wednesday that there's been a "massive spike" in recent days in the number of digital certificate revocations reported via the Certificate Revocation Lists (CRLs). This indicates that businesses are reissuing digital certificates that were in place before they patched OpenSSL.

"The spike is so large that we initially thought it was a mistake, but we have since confirmed that it's real! We're talking about over 50,000 unique revocations from a single CRL," Stanford said. "This is by an order of magnitude the largest spike in revocation activity seen in years, according to our current data."

However, one related cause for concern is that the volume of revoked certificates being reported by various CRLs may be so large that, at least in the short term, servers won't be able to keep up with it.

4. Site assessment: Which remain vulnerable?
Which sites are still vulnerable to Heartbleed? Multiple organizations have created tools -- such as the LastPass Heartbleed checker and the Firefox plug-in from proactiveRISK -- to enable consumers to identify which of the sites they use might be vulnerable or have been vulnerable. Other sites are maintaining lists of vulnerable sites and tracking when they've been updated.

When it comes to using website assessment tools, however, you should take their findings with a grain of salt, since their accuracy relies in part on site administrators self-reporting some data. "These checkers will tell you when a site has updated its SSL certificate only if the date that the new SSL certificate was employed was updated as well," Ashley Thurston, community manager at the password manager Dashlane, said in a blog post. "But not all SSL certificate providers updated that date when they rolled out the new certificates. In short, looking at that date is not enough."

5. Users: When to update passwords
For website users, the immediate concern -- and one of the few aspects of the situation over which they have direct control -- concerns their passwords. The prevailing advice at the moment is to change all your passwords, starting with the most critical sites, such as online banking and email accounts. After a vulnerable site has updated its digital certificates, change the passwords again, and that Heartbleed inoculation should be complete.

6. Android: Heartbleed hits 4.1.1, custom 4.2.2
Some Android users are also at risk, and they will have to wait for updates from their device manufacturer or carrier. But who, exactly, is at risk? The mobile security firm Lookout created a Heartbleed Detector, so Android users can assess whether their version of the operating system is vulnerable.

Lookout said via email Tuesday that, of the 102,000 Android users who had used the scanning tool to date and agreed to share their results, only 4% had devices that were vulnerable. Overall, 86% of users running Android 4.1.1 were affected, while 5% of users running 4.2.2 were affected. "This suggests 4.2.2 is patched, and those affected are running custom ROMs."

7. Android apps connect to vulnerable servers
Many Android apps are also at risk from Heartbleed, because they connect to vulnerable servers. Last week, Trend Micro reported finding 1,300 apps on Google Play -- which offers 390,000 apps -- that connected to vulnerable servers, including 15 bank-related apps, 39 payment-related apps, and 10 online shopping apps, as well as "several popular apps" on the IM and mobile-payment front. By Sunday, Trend Micro had reported finding 7,000 Google Play apps that connected to vulnerable servers.

In addition, the company found 273 apps available via Google Play that "are bundled with the standalone affected OpenSSL library, which means those apps can be compromised in any device."

8. Oracle: 20 applications may be vulnerable
For businesses that use Oracle, the company warned in a security advisory Wednesday that six of its applications are vulnerable to Heartbleed and have been patched. Those applications are Oracle Linux 6, MySQL Enterprise Monitor, MySQL Enterprise Server (version 5.6), Oracle Communications Session Monitor Suite (3.3.40, 3.3.50), Oracle Mobile Security Suite, and some instances of Solaris 11.2.

Oracle also said it's still investigating 14 other applications that may be vulnerable to Heartbleed. They range from ATG Products and MySQL Connector/C++ to Oracle Service Bus and Oracle SOA Suite. The company hasn't committed to a timeline for releasing further required patches.

9. VMware
By comparison, VMware has said that 27 of its products will need a Heartbleed patch, and it has promised to ship all related updates by April 19. After being patched, affected products shipped with OpenSSL 1.0.1 will need to have their digital certificates replaced and their passwords reset. The affected products include NSX for Multi-Hypervisor Manager (4.0.x and 4.1.x), vCenter Server 5.5, VMware vCloud Automation Center 6.x, and VMware vCloud Networking and Security 5.5.1.

10. Vendors still reviewing products for Heartbleed
As Cisco's security warning makes clear, many vendors don't yet know how many of their products might be vulnerable to Heartbleed. That's going to create ongoing confusion for enterprise patch managers, compounded by the fact that there's no single, reliable source of information so far about Heartbleed bugs, in part because information about the vulnerability has rapidly become public knowledge.

"The lack of coordination preceding the disclosure of the vulnerability means that everybody is now playing catch-up, trying to contain the damage," Kasper Lingaard, head of research at Secunia, said via email. "Smaller vendors with only a few vulnerable programs in their portfolio, only have a few patches to roll out. But for bigger vendors, like Cisco, IBM and HP, it's a very different story."

11. More infrastructure: Scope still unclear
Furthermore, when it comes to enterprise infrastructure, some security experts say it may take businesses at least another 24 months to patch every last vulnerable internal web server and SSL-enabled service, which may range from FTP and VoIP phones to printers and VPN servers and clients, including OpenVPN. Of course, that timeline assumes businesses correctly inventory and identify all vulnerable systems in the first place.

As that suggests, fixing Heartbleed won't be cheap. Some experts say the cleanup costs, including patching systems and reissuing digital certificates, could run to hundreds or even thousands of dollars per server.