Test: Most Web Application Scanners Missed Nearly Half Of Vulnerabilities

Most Web application scanning tools miss vulnerabilities and generate false positives on their own public testing sites, according to a recent test of some of these products.

Larry Suto, an application security consultant, tested the Web app scanners for accuracy and false positives as well as the time it took with each to get the best possible results, including running, reviewing, and supplementing the results from the scans. He tested Acunetix, IBM's AppScan, Portswigger.net BurpSuitePro, Cenzic's Hailstorm, HP's WebInspect, NT Objectives' NTOSpider, and Qualys' managed scanning service.

Suto says what surprised him most about what he found in the tests was how the tools didn't catch vulnerabilities and threw false positives when scanning their own test Websites. "I think the report shows that while these tools are very helpful, one should not rely on them exclusively for security," he says.

He found that, overall, the tools missed nearly half of the vulnerabilities on the sites -- even when they were fully "trained," or tuned, rather than set to point and scan the sites. And the big-name products weren't necessarily the most accurate: NTOSpider fared the best overall, with a 94 percent accuracy rating after being trained, followed by Hailstorm, with 62 percent, AppScan (55 percent), Acunetix (47 percent), BurpSuitePro (36 percent), and WebInspect (33 percent). Qualys' service caught 28.5 percent, but Suto notes that it's pure "point and shoot" scanning.

The test (PDF) counted specific types of vulnerabilities: authentication bypass or brute forcing, SQL injection/blind SQL injection, cross-site scripting/persistent cross-site scripting, command injection, XPath injection, SOAP/AJAX attacks, CSRF/HTTP response splitting, arbitrary file upload attacks, remote file include (PHP code injection), and application errors.

Jeremiah Grossman, founder and CTO of WhiteHat Security, says the culprit with accuracy is the scanner's crawling engine. "The crawling engine is most important," Grossman says. "When scanners miss vulnerabilities, it's almost always the case that they didn't find the link they had to test that one vulnerability."

Suto found "moderate" improvements in accuracy when the tools were "trained," except for the case of Hailstorm, which went from 39.6 percent accuracy with "point and shoot" to 62 percent after being trained.

The fastest scanners (in order) were: BurpSuitePro, Qualys, WebInspect, NTOSpider, Appscan, Hailstorm, and Acunetix. When factoring in man-hours, BurpSuitePro and Qualys also came out on top.

Suto's findings in his Web app scanner tests reflect the challenges facing Web application security, as Web applications are often riddled with flaws. "This is a lot harder problem than network scanning. These results should cause security professionals to have significant reason for concern if they are relying on one of the less accurate tools. There is a good chance that they are missing a significant number of vulnerabilities," he wrote in the report.

Wolfgang Kandek, CTO at Qualys, says while a network application scanner detects an old version of an app and then pushes a patch, Web application scanning is much more complicated: "With a Web application scan, you have to go back to your developer and tell him or her, 'Here's the problem,' " he says. There's the element of re-education that comes in the wake of a Web vulnerability discovery, he says.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Dark Reading's editors directly, send us a message.