Can automated penetration testing replace humans?

In the past few years, the use of automation in many spheres of cybersecurity has increased dramatically, but penetration testing has remained stubbornly immune to it.

While crowdsourced security has evolved as an alternative to penetration testing in the past 10 years, it’s not based on automation but simply throwing more humans at a problem (and in the process, creating its own set of weaknesses). Recently though, tools that can be used to automate penetration testing under certain conditions have surfaced – but can they replace human penetration testers?

How do automated penetration testing tools work?

To answer this question, we need to understand how they work, and crucially, what they can’t do. While I’ve spent a great deal of the past year testing these tools and comparing them in like-for-like tests against a human pentester, the big caveat here is that these automation tools are improving at a phenomenal rate, so depending on when you read this, it may already be out of date.

First of all, the “delivery” of the pen test is done by either an agent or a VM, which effectively simulates the pentester’s laptop and/or attack proxy plugging into your network. So far, so normal. The pentesting bot will then perform reconnaissance on its environment by performing scans a human would do – so where you often have human pentesters perform a vulnerability scan with their tool of choice or just a ports and services sweep with Nmap or Masscan. Once they’ve established where they sit within the environment, they will filter through what they’ve found, and this is where their similarities to vulnerability scanners end.

Vulnerability scanners will simply list a series of vulnerabilities and potential vulnerabilities that have been found with no context as to their exploitability and will simply regurgitate CVE references and CVSS scores. They will sometimes paste “proof” that the system is vulnerable but don’t cater well to false positives.

Automated penetration testing tools will then choose out of this list of targets the “best” system to take over, making decisions based on ease of exploit, noise and other such factors. So, for example, if it was presented with an Windows machine which was vulnerable to EternalBlue it may favor this over brute forcing an open SSH port that authenticates with a password as it’s a known quantity and much faster/easier to exploit.

Once it gains a foothold, it will propagate itself through the network, mimicking the way a pentester or attacker would do it, but the only difference being it actually installs a version of its own agent on the exploited machine and continues its pivot from there (there are variations in how different vendors do this).

It then starts the process again from scratch, but this time will also make sure it forensically investigates the machine it has landed on to give it more ammunition to continue its journey through your network. This is where it will dump password hashes if possible or look for hardcoded credentials or SSH keys. It will then add this to its repertoire for the next round of its expansion. So, while previously it may have just repeated the scan/exploit/pivot, this time it will try a pass-the-hash attack or try connecting to an SSH port using the key it just pilfered. Then, it pivots again from here and so on and so forth.

If you notice a lot of similarities to how a human pentester behaves, you’re absolutely right: a lot of this is exactly how pentesters (and to a lesser extent) attackers behave. The toolsets are similar and the techniques and vectors used to pivot are identical in many ways.

So, what’s different?

First of all, the act of automation gives a few advantages over the ageing pentesting methodology (and equally chaotic crowdsourced methodology).
The speed of the test and reporting is many magnitudes faster, and the reports are actually surprisingly readable (after verifying with some QSA’s, they will also pass the various PCI DSS pentesting requirements).

No more waiting days or weeks for a report that has been drafted by human hands and gone through a few rounds of QA before being delivered. This is one of the primary weaknesses of human pen tests: the adoption of continuous delivery has caused many pen test reports to become out of date as soon as they are delivered, since the environment on which the test was completed has been updated multiple times since, and therefore, had potential vulnerabilities and misconfigurations introduced that weren’t present at the time of the pen test. This is why traditional pentesting is more akin to a snapshot of your security posture at a particular point in time.

Automated penetration testing tools get around this limitation by being able to run tests daily, or twice daily, or on every change, and deliver a report almost instantly.

The second advantage is the entry point. A human pentester may be given a specific entry point into your network, while an automated pentesting tool can run the same pen test multiple times from different entry points to uncover vulnerable vectors within your network and monitor various impact scenarios depending on the entry point. While this is theoretically possible with a human it would require a huge budgetary investment due to having to pay each time for a different test.

What are the downsides?

1. Automated penetration testing tools don’t understand web applications – at all. While they will detect something like a web server at the ports/services level, they won’t understand that you have an IDOR vulnerability in your internal API or a SSRF in an internal web page that you can use to pivot further. This is because the web stack today is complex and, to be fair, even specialist scanners (like web application scanners) have a hard time detecting vulnerabilities that aren’t low hanging fruit (such as XSS or SQLi)

2. You can only use automated pentesting tools “inside” the network. As most exposed company infrastructure will be web-based and automated pentesting tools don’t understand these, you’ll still need to stick to a good old-fashioned human pentester for testing from the outside.

To conclude, the technology shows a lot of promise, but it’s still early days and while they aren’t ready to make human pentesters redundant just yet, they do have a role in meeting today’s offensive security challenges that can’t be met without automation.