<i>Tech Insight</i>: How to Pick The Right Web Application Vulnerability Scanner

The mistake most people make when they first buy a Web application vulnerability scanner is to assume it's a simple point-and-click tool.

"It's not like network scanning where you go to an IP address and scan the network," says Danny Allan, director of security research for IBM Rational Software, which sells the AppScan vulnerability scanner. "This is not just a point-and-click product."

Web application vulnerability scanning -- also known as "black box" testing (as opposed to source-code scanning, or white-box testing) -- touches on various levels, transactions, and interactions associated with a Web application. And it requires an experienced hand to run it in order to get the most out of the process of detecting security flaws in Web applications, security experts say.

"The people who are running the scanner matter a lot more than the scanner itself. These are not simple hammers anyone can use. They require the operator to have a significant level of Web security knowledge," says Jeremiah Grossman, CTO of WhiteHat Security, a Web security services firm.

Another misconception about these devices is that the more vulnerabilities they find, the better they are. "Many people go by vuln counts in Web scanners, which is incorrect," notes Caleb Sima, CTO of the application security center at HP, which sells the WebInspect Web app scanner. That's because some products lump together multiple iterations of a specific vulnerability. If one scanner finds 12 SQL injection flaws, and another finds five, it doesn't mean the second one is necessarily missing more bugs, Sima says.

Consider, then, how a scanner counts vulnerabilities rather than how many times it finds SQL injection bugs. Eve more important, IBM's Allan says, is the underlying coding problem that caused the vulnerability. "You may have one cross-site scripting vulnerability and 80 different ways to exploit it," he says. "You need to focus not on the [vulnerability] issue, but on why it happened...that helps prevent security issues from happening again in the future."

That said, it's not so simple to compare these products head-to-head. A European security researcher's recently released test results comparing three major Web app scanners highlighted those challenges given the differences in their approaches. Anantasec performed vulnerability scans against several applications using Acunetix WVS version 6.0, IBM Rational AppScan Version 7.7.620 Service Pack 2, and HP WebInspect Version 7.7.869. He concluded that Acunetix performed the best overall, but as a second layer of analysis, he also used Acunetix's AcuSensor, which looks at source code using a form of white-box testing. So it wasn't actually an apples-to-apples comparison, experts say.

The three products posted mixed results in finding specific vulnerabilities among different applications in the tests; for instance, in some apps one tool would miss XSS flaws, while in others it would find most of them. "Web applications can be very complex, and there are a lot of reasons that would cause a scanner to miss a vulnerability," Anantasec says. "Some of the reasons are poor crawling capabilities, bad JavaScript parsing, inconsistent scanning, or just bugs."

So what should you look for when selecting a Web application vulnerability scanner? IBM's Allan says to first look at how well they test for known vulnerabilities, conceding that most products are fairly equal in the regard. "Most products have similar capabilities in the testing," he says.

Another important feature is the ability to maintain your login state during a scan so that if the person running the scan gets logged out during a test, he doesn't have to start all over again. "The ability to login and maintain login state effectively is hugely important because if the scanner cannot [do so], the scan is invalid because the functionality [would remain] untested," WhiteHat's Grossman says.

In addition, look for whether the product supports JavaScript and Flash so you can scan for flaws in these application types. That's key given the growth of Web 2.0-based apps, experts say, but is still a weak area for scanners. "While these scanners can technically assess XML Web services and identify vulnerabilities in Flash-related software, no product has proved to be anywhere close to comprehensive," says Grossman, who cautions not to blindly trust the results of that part of the scan.

It helps to know the features and functions of the scanner before you test it so you can get the most out of your test -- and prevent any problems. Don't just rely on a demo test the vendor provides, either. Test it against your own Website apps.

Not all products have all of their features on by default, so learn about the options and don't just enable them all without considering the impact on your specific application. "Turning on all security tests and scan configuration can be damaging," IBM's Allan notes.

A retailer testing IBM's AppScan, for example, turned on all of its options -- including "execute JavaScript" -- which inadvertently crashed its email server. "They executed JavaScript...[The application] had a 'mail to' form and sent thousands of requests to the mail server and crashed it," he says. "You need to see if these are the right options [for scanning your application]. Thankfully, it was a fairly minor incident, but it shows that scan configuration needs to be considered carefully -- especially in production systems."

Meanwhile, HP's Sima says many of his company's enterprise customers are going to the next level -- performing recurring Web scanning. "This is where the majority of scanning is done on an after-production scale," he says. "This is a scan of a high-level policy on all Web-facing properties that hold the company's branding, including off-site. It allows the company to identify its risk and quickly identify glaring holes in things it may or may not control, but can cause damage from a branding perspective -- or worse."

One aspect security experts agree on is that black-box testing alone isn't enough. "Black-box testing should be used together with a sensor technology like AcuSensor, and you should also add source-code review into the mix," Anantasec says.

Nick Selby, vice president and research director with The 451 Group, says to look out for false positives and false negatives with these tools, as well.

"At their most basic, Web application vulnerability assessment tools can give you a basic snapshot of known vulnerabilities in Web applications, and some kind of explanation of what it finds, what the severity of a given vuln is, and tips and suggestions on how to fix them," Selby says. "Because this is a very fluid environment, though, false positives and false negatives abound. That's why we suggest vulnerability scanning services in addition to the software -- many firms including White Hat and Cenzic offer these [services]."

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message