The State of Application Penetration Testing

Misconfiguration ranks as the most common type of vulnerability discovered in real-world penetration tests, according to a newly published study.

In client engagements last year, pen-testing-as-a-service provider Cobalt found mostly misconfiguration, followed by cross-site scripting (XSS), authentication and session, exposure of sensitive data, and access control-type vulnerabilities in applications.

Finding flaws is one thing, but fixing them is another. Redirect and forward-type flaws sit unresolved the longest of any category, 41 days, while server-side request forgery, sensitive data exposure, SQL injection, and others, including business logic, get fixed most rapidly.

Caroline Wong, vice president of security strategy for Cobalt and co-author of the Pen Test Metrics Study, says misconfiguration flaws are a sign of the times. "What that tells me is that so much of security vulnerability comes not necessarily from the code we were writing, and actually may have to do with other software and infrastructure components our software depends on in order to run," she says.

"That says … organizations are making huge use of cloud services and relying on others to do their settings for them. Maybe it's something they consider to be someone else's problem, and maybe in the past they didn't depend on third parties so they didn't consider [those] security settings."

Application penetration testing, unlike vulnerability assessment, is not exactly standard practice for most organizations today. Pen testing traditionally was associated with network security, but with the emergence of secure software development lifecycle (SDL) programs, more organizations are starting to opt for a white-hat hack of their apps. "I see a lot of organizations do one application pen test a year because of PCI, or HIPAA, or a customer asking them for one," Wong says. But more organizations are starting to opt for app pen tests to "do the right thing" in their secure development practices now, she says.

The most common software security program typically includes developer training and a penetration test to get a pulse on the state of their applications' security. Wong says organizations launching appsec for the first time go with a pen test to get them started.

"What's the biggest bang for their buck to start their program? 'Show me and my organization if we have real security issues, and use that information to get to the next level to justify further investment,'" she explains. "I find that pen testing is a very common first step when they first start thinking about appsec."

While a vulnerability assessment scans for and identifies flaws, a penetration test goes deeper, manually exploiting vulnerabilities to see how an attacker could abuse them, for example.

Most organizations typically focus on vulnerability scanning. "But by and large there are pockets who do true pen testing. I think there's a larger segment that does quasi-pen testing and not full, in-depth testing," says Kevin Greene, a software assurance evangelist. "I'm not sure all folks understand" app pen testing, he notes.

Ideally, in addition to an SDL check, organizations should run regular vulnerability assessments and pen tests to get "wider coverage" of the attack surface, he says.

Gary McGraw, vice president of security technology at Synopsys, says app pen testing is employed by 87% of all firms involved in the BSIMM8 software security maturity study. BSIMM  (Build Security In Maturity Model) reports on how more than 100 major companies from a range of vertical markets measure up with their software security development lifecycles. "Pen testing is a good 'smoke test' which can help uncover major problems," McGraw says. "Automated pen testing is available as a service and can be used to cover an entire application portfolio."

But pen testing alone doesn't make a software security program, he notes. "Pen testing is the third most important software security practice after code review with a static analysis tool and architectural risk analysis," McGraw says.  

Not All Apps Getting Tested

Cobalt's study also includes new data from a survey of security, management, operations, developers, and DevOps specialists. Turns out most aren't pen testing all of their apps: just 24% say they pen test 67% to 100% of their apps, while 35% test one to 33% of their apps, and 31% say they test anywhere from 34% to 66% of their apps.

While the best practice is to pen test critical apps once every quarter, most (32%) of the respondents in the survey say they only do so annually; 16%, semiannually; 12%, quarterly; 12%, not at all yet; and 7%, more than five times a year.

More than 30% say they pen test their apps when they add a new feature or patch. Some 26% pen test their apps on an ad hoc basis, 25% at the time of a new release, and 22% when a customer requests it.

Some 46% say they would pen test more apps, but it's too expensive. That's a common deterrent, as well the expertise required for pen testing. "It requires a really skilled individual" with the expertise to know what to probe and attack, according to Greene.

And once the results come in from a pen test, they require action. "I have met organizations for whom the reason they don't do more pen tests is because they are still trying to figure out how to fix the results from their first pen test," Wong says.

"The biggest challenge of pen testing apps is finding the right people," which can be costly, she notes.

Related Content:

Black Hat Asia returns to Singapore with hands-on technical Trainings, cutting-edge Briefings, Arsenal open-source tool demonstrations, top-tier solutions and service providers in the Business Hall. Click for information on the conference and to register.