Outer Limits of IPS

5:55 PM -- When traffic spikes, are you actually under attack? Will deployed intrusion prevention hardware start blocking traffic to your site? Is it safe to block that traffic -- especially automatically?

These are the questions you should be asking yourself if you are going to deploy an intrusion prevention system (IPS).

Several years ago Richard Stiennon -- then still with Gartner -- told the world that the intrusion detection system (IDS) was dead. He was talking about the fact that companies don't recoup their costs by deploying something that simply monitors that they are under attack. Rather, he argued, they should instead invest in a smarter solution that actually does something to prevent the attack, like -- an intrusion prevention system (IPS) or hardware to combat distributed denial-of-service (DDOS) attacks.

IPSes are typically no more than glorified rules engines tied in with a firewall. There are different versions; some that send packets to kill the connection (like the great firewall of China that protects the entire country from bad words, like the phrase that's a form of Tai Chi with religious implications). Others simply drop the packets. In the end, the intended effect is the same: The connection with the malicious traffic is disrupted. But is that what you really want? What is triggering these rules?

There are two types of detection, anomaly- and rules-based. Rules-based says that the malicious traffic must perform a particular function that matches what's on the rules engine in order to be blocked. Anomaly is based on the premise that traffic patterns tend to follow a particular pattern. If traffic ever spikes above normal it's an anomaly and it should be stopped.

But here's how each type of detection can easily fail. In the case of the great firewall of China, they send packets in each direction to shut down the connection if they find a bad word. But if someone were to try and encode even vaguely the bad word by reversing the text, using pig latin, or any of a thousand other techniques, then the rules engine would not fire. There are other problems with China's method, in that if you simply ignore the packets they send to shut down the connection, you can continue to route packets. A flawed solution, indeed.

Anomaly detections only detect when an action is performed that should not happen. In the case of a cross-site request forgery, it is trivial to get a valid user to perform an action which then shuts down that connection for that user (not the attacker). If the attacker can get a search engine to follow a link to a function that it should not attempt to go to, your IPS could actually end up blocking the search engines from spidering your site, which hurts your ability to get traffic to your company. This same problem exists for rules engines as well.

However, there is another issue with anomaly detection. Let's pretend you are Victoria's Secret. All year long your traffic is a low rumble. But once a year your traffic spikes so high that any anti-DDOS engine could not ignore it. Do you really want to prevent millions of viewers from watching your online fashion show?

It looks and smells like a denial-of-service attack, but yet it is one of the critical parts of doing business. Granted DDOS and IPSes are trying to solve two different issues, but this is a good explanation of how anomaly detection can create huge false positives.

While both anomaly detection and rules engines have their own unique issues, both have their uses. I wouldn't recommend ditching your IPS dreams of a safe future. But don't hold your breath. The technology has a long way to go before it is capable of that subtle balance between security and invasiveness.

— RSnake is a red-blooded lumberjack whose rants can also be found at Ha.ckers and F*the.net. Special to Dark Reading