Debugging The Myths Of Heartbleed

Now that IT organizations across the globe have had time to recover from the recent Heartbleed flaw, what can we learn from this incident? The vulnerability was discovered in an OpenSSL library used by thousands of websites on public and private networks and had gone unnoticed for years. Attackers could force a web server to reveal data from inside an SSL session, completely bypassing encryption. As if that weren't bad enough, initial reports claimed that the Heartbleed attack on the TLS/DTLS heartbeat extension occurred "without a trace."

It would not be an understatement to characterize Heartbleed as one of the creepiest security vulnerabilities ever to lurk across the Internet. Here's a quick summary of its timeline:

What's in this bug and what's at stake
Let's be clear: this isn't a flaw in SSL/TLS or in the heartbeat extension (RFC 6520). The vulnerability exists within the OpenSSL implementation of the extension. Heartbleed exploit code allows attackers to force a web server to reveal 64 KB chunks of certain memory regions by overflowing a buffer. While it isn't possible to predict what might be revealed, successful attacks have obtained session keys, passwords, and other information that should normally remain confidential.

Finding the hidden evidence
Does Heartbleed really wreak its havoc while leaving nary a trace? The media and many technical sites seemed convinced of this, but some of us were skeptical. The Heartbleed attacks surely leave some evidence behind: packets. Packets almost always tell a detailed story of what has really happened, including in the case of Heartbleed. The trick, of course, is to have the packets.

It's true that a server attacked with a Heartbleed exploit is unlikely to reveal any evidence. Stored packets, meanwhile, do tell the story of a successful Heartbleed exploit even after the adversary has stopped an active attack.

Detecting a prior Heartbleed exploit
Continuous monitoring of a network can reveal active Heartbleed attacks. But even more importantly, with a sufficiently large rolling buffer of packet capture data, it becomes possible to look back in time, before the public disclosure of the Heartbleed vulnerability. An investigation of this data may reveal whether an actual exploit of vulnerable servers has occurred.

A Berkeley Packet F (BPF) placed in the network can automatically flag larger-than-normal TLS heartbeat responses from servers. Wireshark, tcpdump, and other tools can analyze the captured packets for confirmation of an attack.

Why BPF?  BPF engines are fast -- something of a requirement given the sheer amount of traffic passing through modern networks. Important for the use case here, a BPF capture is a common format for packet processing, understood by the majority of operating systems and packet-analyzing software. The wide availability of BPF is the main reason it can become easy to detect Heartbleed attacks.

BPF engines are available for Linux, for Mac OS, for Windows (via WinPcap), and can be placed on cloud computing instances. BPF and tcpdump are present on most network appliances (such as firewalls, load balancers, and application delivery controllers) and often accessible through an administrative console for troubleshooting purposes. And, of course, packet analysis and storage engines in products designed for supporting network performance management almost all support BPF.

As a result, many individuals in the technical community have responded with plenty of resources to develop a BPF filter appropriate for detecting Heartbleed attacks. Most of these filters examine traffic from port 443/tcp (the default HTTPS port). The usual size threshold is 69 bytes; this can be adjusted upwards or downwards to reduce false positives or false negatives if necessary.

Final thoughts
Having a nimble awareness of the data in your network, a basic understanding of how secure services should normally operate, and the ability to investigate anomalies can inoculate you from the unavoidable hype. Packets do not lie -- but you have to capture them to reveal their truths.

A large rolling buffer of packet capture data establishes an ideal forensics basis. From this, you can determine what has actually occurred during the time when vulnerability exploits run wild. Certainty is always better than mere speculation over hypothetical breaches in security.