Automated Pen Testing: Can It Replace Humans?

Cybersecurity automation has increased dramatically in the past few years, but penetration testing has remained stubbornly immune. While crowdsourced security evolved as an alternative to pen testing in the past 10 years, it's not based on automation but simply throwing more humans at a problem (and, in the process, creating its own set of weaknesses). Recently though, automated pen-testing tools have advanced to a point where they are usable under certain conditions. This begs the question: Can these tools replace human pen testers?

I've spent a great deal of the past year testing these tools and comparing them in like-for-like tests against human pen testers, but the caveat is that these automation tools are improving at a phenomenal rate, so this analysis may already be out of date before you read it.

How Automated Pen Testers Work
These tools "deliver" a pen test by using either an agent or a virtual machine (VM) that simulates the pen tester's laptop and/or attack proxy plugging into your network. The pen testing bot then performs reconnaissance on its environment by doing identical scans as a human would do — running a vulnerability scan with a preferred tool or just a ports-and-services sweep with Nmap or Masscan. Once the automated tools have established where they sit within the environment, they will filter through what they've found. This is where the similarities to vulnerability scanners end.

Vulnerability scanners simply list a series of vulnerabilities and potential vulnerabilities they find — with no context about their exploitability. They simply regurgitate Common Vulnerabilities and Exposures (CVE) references and Common Vulnerability Scoring System (CVSS) scores. They sometimes paste "proof" that the system is vulnerable — but they don't cater well to false positives. The automated pen-testing tools then choose the "best" system from these targets to take over, making decisions based on ease of exploit, noise, and such factors. For example, if the bot finds a Windows machine that is vulnerable to EternalBlue, it may favor this over brute-forcing an open SSH port that authenticates with a password, as it's a known quantity and a much faster and easier exploit.

Once the tool gains a foothold, it will propagate through the network, mimicking how a human pen tester or attacker would do it. The difference is that it installs a version of its own agent on the exploited machine and continues its pivot from there. It starts the process again from scratch, but this time it forensically investigates the machine to give it more ammunition to continue its journey through your network. This is where it could dump password hashes or look for hard-coded credentials. It then adds this to its repertoire for the next round. While previously it might have just repeated the scan/exploit/pivot, this time it will try a "pass the hash" attack or connect to an SSH port using the key it just pilfered. Then, it pivots again and so on.

If you notice a lot of similarities to how human pen testers behave, you're absolutely right — a lot of this is exactly how pen testers (and to a lesser extent, attackers) behave. The tool sets are similar and the techniques and vectors used to pivot are identical in many ways. So, what's different?

Advantages of Automated Pen Testing
Automation offers a few advantages over the aging pen-testing methodology (and the equally chaotic crowdsourced methodology).

First, the speed of the test and reporting is magnitudes faster, and the reports are surprisingly readable. (After conferring with some Qualified Security Assessors, I've verified they will pass the various PCI-DSS pen-testing requirements.) No more waiting days or weeks for a report drafted by human hands and a few rounds of QA before it's delivered into your hands.

This is one of the primary weaknesses of human pen tests today — continuous delivery means many reports are out of date before they are delivered. The environment has been updated multiple times since the test, which introduces new potential vulnerabilities and misconfigurations that weren't there during the pen test. This is why traditional pen testing is really a snapshot of your security posture at a point in time.

Automated pen-testing tools get around this limitation by running tests daily, twice daily, or on every change, and delivering a report almost instantly. This means you can pen test your environment and detect potentially exploitable configuration changes on a daily basis, rather than relying on a report delivered weeks later.

Automation's second advantage is the entry point. While you may give a human pen tester a specific entry point into your network, an automated tool can run the same pen test multiple times from different entry points to uncover vulnerable vectors and monitor various impact scenarios depending on the entry point. While this is theoretically possible with a human, it would require a huge budget to pay each time for a different test.

Disadvantages of Automated Pen Testing
Automated pen-testing tools do have downsides. First, they don't understand web applications — at all. While they will detect something like a web server at the ports/services level, they won't understand that you have an insecure direct object reference (IDOR) vulnerability in your internal API or a server-side request forgery (SSRF) in an internal webpage that a human pen tester can use to pivot further. This is because today's web stack is complex, and even specialist scanners (like web application scanners) have a hard time detecting vulnerabilities that aren't low-hanging fruit (such as XSS or SQLi).

This leads to another weakness in automated pen-testing tools: You can only use them "inside" the network. As most exposed company infrastructure is web-based, and automated pen-testing tools don't understand these, you'll still need to stick to good ol' fashioned humans for pen testing from the outside.

Where Things Stand
This technology shows a lot of promise, but it's early days. While it can't make human pen testers redundant just yet, it has a role in meeting today's offensive security challenges that can't be met without automation.