Bug bounties are cheaper than hiring full-time bug hunters

Software companies that have instituted bug bounties are on the right track, a recently published report by researchers of the University of California, Berkeley computer science department has shown.

Vulnerability rewards programs (VRPs) are 2 to 100 times more cost-effective than hiring expert security researchers to find vulnerabilities, they say, and by comparing the Chrome and Firefox VPRs, they have pointed out why the former is more effective than the latter.

In order to perform the analysis, they took into consideration the number of paid out bounties and the total amount of money they paid out in the last three years, and the difference was immediately striking.

While Google’s program has awarded 501 bounties and a total of $580,000 to the successful bug hunters, Mozilla’s paid out nearly the same amount ($570,000) in only 190 bounties. So, what’s the catch?

For one, the two companies’ VPRs have different rules and rewards. While Mozilla offers a fixed amount ($3,000) for each vulnerability, Google provides different rewards depending on the severity of the bug – and a clear guide- line for deciding severity.

Secondly, the number of contributions by external researchers to Chrome has nearly caught up with the number of internal contributions, but for Firefox that number has consistently been far lower than that of internal contributions.

“Mozilla’s VRP also qualitatively differs from the Chrome Google program,” they also pointed out. “First, Mozilla awards bounties even if the researcher publicly discusses the vulnerability instead of reporting it to Mozilla. Second, Mozilla also explicitly awards vulnerabilities discovered in ‘nightly’ (or ‘trunk’) versions of Firefox. In contrast, Google discourages researchers from using ‘canary’ builds and only awards bounties in canary builds if internal testing would miss those bugs.”

Finally, Google also sponsors a very-well known exploit bounty program fueled by the researchers competing in its Pwnium contest. While the considerable prizes awarded during it were not taken into consideration for this study, the exposure that Google gets from it definitely was.

Among the things that they believe to have discovered during their research are these:

• As the number of researchers looking for vulnerabilities increases, so does the diversity of vulnerabilities discovered.
• In the Chrome VRP, the majority of the rewards are for only $500 or $1,000. Larger rewards are infrequent. “Much like the lottery, a large maximum payout ($30,000 for Chrome), despite a small expected return (or even negative, as is the case of anyone who searches for bugs but never successfully finds any) appears to suffice in attracting enough participants,” the researchers concluded.”
• It pays better to contribute to multiple VRPs instead of just one, and successful independent security researchers “bubble to the top”, making it more likely for them to be offered a full-time job.

“We find that VRPs appear to provide an economically efficient mechanism for finding vulnerabilities, with a reasonable cost/benefit trade-off. We therefore recommend that more vendors consider using them to their (and their users’) advantage,” they advised.

“The cost/benefit trade-off may vary for other types of (i.e., non-browser) software vendors; in particular, the less costly a security incident is for a vendor, the less useful we can expect a VRP to be. Additionally, we expect that the higher-profile the software project is (among developers and security researchers), the more effective a VRP will be,” they concluded.