Automated Pen Testing Is Improving — SlowlyAutomated Pen Testing Is Improving — Slowly

COMMENTARY

When automated pen-testing tools appeared a few years ago they prompted an interesting question: How close are they to replacing human pen testers? While the short answer was "not that close — yet," they definitely had potential and were worth keeping an eye on.

As I've just had the chance to review the latest iteration of these tools, it's interesting to see how they've evolved and how close are they now are to replacing the human pen tester for offensive security work.

When I test an automated pen tester, I compare it with a human one, in terms of speed, capability, and capacity, as well as output (i.e., the resulting report). The big problems earlier automated pen testers suffered from included:

How Have Automated Pen Testers Changed Since Then?

New pen testers finally understand Web applications — hooray! They can attack them both from inside and outside the perimeter. This is a welcome development, but they still have teething issues. Due to a very mature market in Web application scanners they would need to be able to both detect vulnerabilities with a low false positive ratio and be able to exploit them to pivot to other assets.

Unfortunately, they don't do this well enough to be distinctive in their own right — they'll find vulnerabilities that are obvious enough, but on a vulnerable box weren't able to detect even blatant SQLi or validate potential XSS vulnerabilities to weed out false positives. There are flashes of brilliance, however. An internal Web endpoint had a file upload vulnerability that was previously undetected by any other tool (this wasn't even found by human pen testers), but overall, it's underwhelming. Today's offerings in Web application scanners will do much better than this.

The second big improvement is cloud environments. As most pen testers will tell you, navigating an on-premises Active Directory-based environment is markedly different from pivoting in a native Amazon Web Services (AWS) environment, as the assets and the exploits you will use are completely different. Privilege escalation now relies on leveraging poorly configured cloud assets to abuse an identity and access management (IAM) role or grab some AWS keys to go further. Naturally, you'll also find the traditional vulnerabilities that include unpatched machines and misconfigured ports and services. Here, again, automated pen-testing tools have evolved, and can navigate and understand these environments. This puts them on par with CNAPP-type offerings, since they aren't bound by the traditional VM- or IP-bound asset.

As the cloud is a relatively new sphere for these tools, they can struggle. Unless they are given an assumed role, they won't find much at all. What's worse, they will flag the fact that they've assumed an IAM role a vulnerability itself — this would be like giving pen testers local admin abilities so they can begin a pen test and them pointing out your security is bad because you've just given them local admin.

Automated pen testers also struggle to enumerate their own network when they are given access — machines that are obviously on the same virtual private cloud (VPC) or virtual LAN (VLAN) will be ignored or scanned haphazardly. This is better than some automated pen testing tools that still don't even work in cloud environments unless they can reach an Active Directory machine. 

Automated Pen Testers' Advantages

All of the other advantages you'd expect from these tools remain, however. They can run through an iteration of a pen test quickly — in a matter of hours if you wish (this is configurable). The reports they produce are top-notch and comparable to any report a human pen tester would produce. If you were to hand this to a qualified security assessor (QSA), they would have a hard time distinguishing the difference.

Naturally, due to their automated nature, you can propagate these on huge environments and repeat them on a daily basis if you wish. This is where automated pen testers leave humans in the dust — no company can repeat daily pen tests on large environments, even with significant budgets, nor would the human team be able to complete it in this time and write up a report with verifiable actions to make it meaningful enough. (Keep one thing in mind: These tools aren't cheap.)

Overall, it's good to see these tools evolve. The rate of change is glacial, but they now understand cloud environments and can target Web applications, though they are still temperamental, costly, and miss a few things. One could argue humans are the same. For now, however, humans maintain the advantage — but they aren't mutually exclusive. Just like crowdsourced security and traditional pen testing, automated pen testing is now another tool that can be layered onto your offensive security testing, where it can help you find the exploits that matter to your organization.