Analyzing Network Chaos Leads to Better DDoS Detection

The Internet is a chaotic medium — packets tend to flow from a uniformly distributed set of sources to a variety of destinations.

Yet during a distributed denial-of-service (DDoS) attack, the chaos suddenly becomes more ordered: A large number of devices send network packets to a limited number of addresses in a small time frame. By analyzing such unusual changes in the entropy of the Internet, a group of researchers with the Pacific Northwest National Laboratory (PNNL) said they can identify 99% of DDoS attacks with only a 2% false-positive rate on average. They compared their method with a set of 10 standard algorithms, which identified only 52% of attacks on average and 62% of attacks in the best-case scenario.

The algorithm — which the researchers dubbed "DDoS attack detection via differential analysis of generalized entropy," or DoDGE — is both more accurate and less prone to falsely identify an attack than other measures, says Omer Subasi, a computer scientist at PNNL and an author of a paper on the topic presented to the IEEE International Conference on Cyber Security and Resilience.

"Under normal circumstances, traffic from senders and to receivers are relatively well distributed, and this level of entropy remains fairly stable," he says. "However, under attack scenarios, we detect an imbalance between the senders and receivers. By quantifying how this changes over time and the degree of change, we are able to identify ongoing attacks."

While ransomware and business email compromise (BEC) attacks tend to garner the most attention from security groups, DDoS attacks continue to be the most impactful for businesses. Over the past four years, DDoS attacks have accounted for the greatest share of security incidents reported by companies, according to Verizon's annual "Data Breach Investigations Report."

Datasets show the difference in entropy between benign traffic (upper left), an attack (upper right), and two flash events caused by soccer matches. Source: Pacific Northwest National Laboratory

Better methods of detection can help businesses respond more quickly to attacks and put in place better countermeasures, says Allen West, a researcher at Akamai.

"Being able to confirm whether a DDoS attack is currently taking place allows defenders to confidently deploy targeted defense mechanisms, such as precise traffic filtering and other DDoS-specific protection services," he says. "It also empowers the target organization to gather more information about the incident that is valuable from an intelligence point of view, which may allow them to deduce the source or reason behind the attacks."

Internet Chaos Is Normal

The most common approach to detecting denial-of-service (DoS) attacks is to create a threshold — a top bandwidth or packet count above which a surge in traffic is considered an attack. The PNNL research instead measures the entropy of network traffic, specifically focusing on how two measures of entropy change: At the target, requests for a specific resource increase during a DDoS attack, leading to less entropy, while the number of sources grows, boosting entropy.

By looking at the small changes over time, the researchers differentiated between surges of legitimate traffic — so-called "flash events" — and actual attacks, says Kevin Barker, a principal investigator at PNNL.

"Only some of the existing work even tries to address this problem of differentiation," he says. "Alternative solutions either use thresholds or are ML/AI-based, which require large data and incur costly training and retraining to adapt."

The ability to quickly differentiate between a true attack and a surge in legitimate traffic due to, for example, a news event or viral content is critical to determining a response, says Akamai's West.

"With a DDoS attack, efforts to identify and block malicious traffic while keeping legitimate traffic will be the main priority," West says. "With 'flash events,' however, different actions can be taken to handle this load as gracefully as possible without taking more aggressive measures."

False Positives Still Need to Fall

The entropy-based detection of DDoS attacks improves significantly with threshold-based methods, with a relatively small rate of misclassifying legitimate content (known as a false positive), according to the researchers. The technique had false-positive rates of less than 7% in all cases and less than 2% on average across 10 real-world data sets.

Yet to be useful in the real world, such techniques need to have a false-positive rate approaching zero, says Patrick Donahue, vice president of product at Cloudflare.

"Over the years we've seen research techniques published that appear to work well in the narrowly defined parameters of the lab but aren't effective or are unable to scale," he says. "For example, the false-positive rates that customers in the real world will tolerate and the sampling rates required to detect at scale often differ materially from what’s acceptable in the lab."

The PNNL researchers stress that their algorithms are adaptive, so the false-positive rate can be minimized by sacrificing some precision in attack detection. Moreover, in real-world scenarios, additional data can be used to augment the basic algorithm.

Because it is relatively lightweight from a computational perspective, the DoDGE algorithm could have benefits for building resilient infrastructure for 5G networks, which are expected to significantly boost the number of connected devices, PNNL's Barker stated in the lab's announcement.

"With so many more devices and systems connected to the internet, there are many more opportunities than before to attack systems maliciously," Barker said. "And more and more devices like home security systems, sensors and even scientific instruments are added to networks every day. We need to do everything we can to stop these attacks."