Crowdsourced pentesting is not without its issues
Crowdsourced security isn’t new anymore, having existed in one form or another as a consumable enterprise service since 2013 with the launch of the main crowdsourced platforms (HackerOne, Bugcrowd and Synack). Slowly but surely, these platforms challenged traditional pentesting practices and started to eat away at their market share. Further platforms and competitors have since launched within the crowdsourced space to compete for a part of this growing market share.
But is crowdsourced security really a panacea to the ills of traditional pentesting or does it create more issues? Before we tackle this let’s cover what the issues of traditional pentesting actually are.
Development cycles and continuous delivery
For companies that utilise pentesting, it is usually a once-a-year exercise. Sadly, this doesn’t keep pace with the speed of development today. Many organizations deploy weekly, daily or are in a continuous delivery methodology, constantly changing their environments and applications and hence potentially introducing vulnerabilities and configuration issues at a constant pace.
A pentest performed on this kind of environment will only produce a snapshot of a security posture at a specific point in time (the generally accepted definition of pentesting). Add to this the time it takes for a report to be drafted, go through QA and delivered to the customer (usually several weeks) and a pentesting report is out of date as soon as it’s delivered to the customer. In that time the customer environment has changed multiple times and is no longer representative of what was tested in the first place.
Time-limit
A commercially imposed limitation but a very important one. Pentesters don’t have the luxury of time, and tests are usually time-limited. A website engagement may typically be assigned 5 days, one day of which is reserved for report writing. This means a pentester doesn’t have time to deep dive into every nook and cranny of the application and will constantly have to make decisions on what to pursue and what to ignore in the time they have allotted.
Skillset
There is a variance in skillset, even among pentesters. Some are better at testing mobile apps, others at testing API security and web applications. Still, the technologies are so varied that you will find variations in skillset even in a small population of specialized pentesters. Add to this the difficulties in hiring skilled staff today (a theme that’s not new in infosec) and you’ll often run into the problem of two different pentesters testing the same application and finding different vulnerabilities. A tactical solution to this has been to “cycle” pentesting suppliers each year but – the pentesting pool of talent being so small and specialized – I’ve witnessed companies ending up with the same pentester two years in a row, but now working for a different company!
Pentester syndrome
Pentester syndrome is making things appear worse than they actually are. A common practice in pentesting reports is to “talk up” issues you’ve found, especially if you couldn’t find anything critical. This is also why no-one’s ever read a pentesting report which says “everything’s ok” – I’ve seen even informational things like a missing Strict Transport Security Header appear as a “medium” vulnerability. This generates unnecessary work chasing down “junk risk”, which will remediate issues in a pentesting report, but not improve your security posture one bit.
Business model
Lastly, there’s a business model disadvantage to having to keep a roster of pentesters on your payroll. You have to pay them a competitive salary, provide them with the licenses for all the equipment they need (e.g., Burpsuite Pro licenses, etc.), as well as sponsor their ongoing training and skillset, send them to conferences and all the baggage associated with full time employees. In a workforce where there is already scarcity, this is expensive and weighs on the bottom line.
How does crowdsourced security solve these issues?
Crowdsourced business models took aim at these issues by adopting a flexible approach to pentesting. There are no dedicated pentesters but a “crowd” of volunteer security researchers that sign up and attempt to find vulnerabilities in an asset. If they find one, they are paid. If they find nothing, they are paid nothing.
The first problem this solves is the “time-limited” aspect of pentesting. No longer do you have just 5 days to try and pick at an application – crowdsourced pentests are typically open-ended, meaning you can spend weeks if not months hunting down elusive, critical vulnerabilities, and this has played out to great effect.
I have my own personal experience with this: as part of a crowdsourced program I once found a critical vulnerability in a multi-billion dollar, Nasdaq-listed company after looking for vulnerabilities for several weeks. This vulnerability allowed the total ownership of all 100 million + customer details (effectively owning all their data). Due to the complexity of the vulnerability, there was no chance any pentester would have had the time to investigate this properly (they relied on traditional pentests in the past, which proved this point).
The second problem about continuous delivery and point-in-time tests is also remediated by this open-ended approach and by having researchers dip in and out of the programs. This has its own issues, which I’ll get into later, but also ensures that despite a constantly changing infrastructure, it is constantly being tested (providing you have a wide and deep enough pool of researchers to draw from).
This leads to the third pentesting issue – skillset and business model. Crowdsourced companies have a huge business model advantage in not having full-time employees. They don’t pay them a salary or even need to pay their material costs. To compensate for skillset issues, they just throw as many bodies at an application as possible, and this will cover all known profiles of the technology stack by sheer numbers. The more eyeballs you have looking at something, the more issues you will find.
Finally, the pentester syndrome issue is resolved by the reward system. If you submit an issue that isn’t really a vulnerability and you don’t provide a proof of concept, then it’s ignored. Worse, your profile will have points deducted for wasting time and/or (in extreme scenarios) be kicked off the platform entirely. The customer gets only actionable vulnerabilities with exploits, not pentesting reports filled with junk risk.
The issues with crowdsourced security today
Taking the above into account, crowdsourced security is not a like-for-like replacement for pentesting today. It still has many issues, some of them intractable due to their business model.
Internal vs external testing
Crowdsourced security tests aren’t suited to testing inside a company perimeter. In a pentest, a consultant physically turns up to the organization’s premises and just plugs his laptop in to begin his tests. In a crowdsourced scenario this isn’t possible since it requires a complex mixtures of VPNs and/or proxies to be set up, and the network has to be able to maintain the load of dozens if not hundreds of users testing at once. This is why the majority of crowdsourced engagements so far have been for web applications, since these can be accessed from anywhere with relatively little cost or complexity.
This extends to any physical testing or testing of IoT devices. While I have participated in crowdsourced engagements where you were sent a physical item (a fitness track for example), this requires investment since every tester involved requires a copy. Add to this that testers are spread out all over the globe and your upfront costs can quickly spiral before you’ve even started the test.
The resource pool is finite
Offensive security suffers from a skills shortage just like every other facet of the information security workforce today. Crowdsourced security, while alleviating this somewhat by expanding the potential pool of testers to an international level, has still hit a brick wall, as there is no endless pool of talent to draw from.
Visit the leaderboard of the main crowdsourced platforms and you’ll find one striking similarity – they’re almost the same. The majority of the testing on all platforms is done by a select group of super-researchers, some of which do it full time. This means the majority of vulnerabilities are actually handled by the same group each time.
While you may read marketing references to having “thousands” of researchers, the reality is that two dozen researchers account for most of the vulnerabilities found on platforms today. This creates a resource problem where crowdsourced companies, many backed by venture capital, require constant growth, and so more customers, and therefore more programs are open for testing. These programs need a corresponding growth in testers, which just isn’t there. Everyone today who wants to participate in a crowdsourced pentest is already doing it. As it’s entirely voluntary, you can see the problem this causes – you cannot force a voluntary workforce to test your new asset when they simply don’t have the bandwidth to do it.
The cost of crowdsourced pentesting
Despite what crowdsourced security companies say, crowdsourced pentesting is not cheap by any standard. A pentest for an external website today will set you back the number of days, multiplied by the daily rate of the consultant.
Let’s take an average and say this is $1200 USD (this can be more or less depending on the pentesting company). For a five day pentest (common for a website) this gives you an average of $6000 USD to test an asset. To get a crowdsourced test first you need a platform fee which is many times that – the fee to actually advertize your pentest on the various crowdsourced platforms. Add to this that you also have to pay out a reward for every vulnerability that’s found so the more vulnerabilities that are discovered the more you pay out, and this means your costs can quickly spiral out of control.
A couple of caveats here: Synack’s (one of the platforms) approach is slightly differently – they only charge a platform fee and all reward payouts are from their own pocket. This cost premium effectively rules out crowdsourced testing for smaller companies due to the barriers for entry being so high.
Federacy is, for now, the only alternative for small/medium businesses. They cater to them by lowering the platform and payouts, but the problem is that the lower the payouts, the less researchers will be attracted to the plaftom.
The gig economy
Probably the most insidious issue is that crowdsourced security effectively propagates an Orwellian version of the gig economy. The gig economy today is more commonly associated with the likes of Uber and Deliveroo – workers forego traditional benefits like pensions and sick pay to choose when and how much they work.
There is one crucial difference though: gig economy workers are actually paid for their labor. If you choose to work as an Uber driver for 10 hours, you will be able to calculate a certain amount of take-home pay. Security researchers engaged in crowdsourced pentesting are not paid for the work, but per found vulnerability, and they can easily spend a day searching for vulnerabilities and find nothing. Finding no vulnerabilities is actually the default result for most security researchers today, and you are paid absolutely nothing for your time.
Not only this, but all the tools you use you must procure yourself. Need a jailbroken iPhone to test that mobile app? You need to provide it yourself. Need a copy of Burp Suite Pro? You need to buy the license yourself. You’re sick? Too bad. Pension? What’s that? This has huge costs savings for crowdsourced security companies since they effectively solve the business model issue pentesting companies struggle with, but introduce exploitation of a workforce as a result.
To conclude, note that both approaches can be complementary, despite their myriad of issues. There is no one solution for offensive security testing, and it’s up to you decide which fits your environment best while keeping the above issues in mind.