Thursday, 13 October 2016

Bounties Bug Me (A Little)

The popularity of Bug Bounties has grown hugely over the last 3-5 years, with start-ups taking advantage of eager VCs and Angel investors who want a piece of the ‘Cyber’ pie. As this niche market has become more mature, growth opportunities for crowd sourced frameworks have peaked and pressures from investors to diversify have amplified. As a result, we now see emboldened assertions from some providers that their programs replace services such as penetration testing. For me, this is quite a stretch and I’m going to discuss the reasons why in this post, with some input from some of the world’s top bounty hunters.

I’d like to state from the off, that this isn’t an attack on Bug Bounties as a notion, but a counter to marketing by organisations in this space that misrepresent what these types of programs can logically provide. 

As a Senior Director at one of the world’s largest pen testing practices, I’m keen to stay as objective as possible, so I decided to reach out to a few prolific bug hunting friends to get their opinions and to keep me honest. Please note that my views are my own and not that of my employer. A huge thanks to Yappare, Shubham Shah and John ‘n0x00’ Carroll for their inputs. In order to give you some background of the quality of their inputs and why you should listen to these guys, I’ve included a quick overview of the contributors to this post. Yappare is currently (at time of writing) ranked 2nd All-time on BugCrowd and is the author of a fantastic blog, that can be found here. John Carroll is currently an AppSec gun-for-hire and one of the original Top 10 bug bounty hunters for Bugcrowd back in 2013, he was also one of the most prolific charity bounty hunters around. John’s blog is really well known for its humour and innovative techniques, check it out here.  Shubham Shah is regarded as one of the most innovative bug hunters in the world, regularly speaking at conferences about his tools and techniques to make the most out of ‘bounty time’, he’s also fairly prolific on HackerOne. Check out his blog here; his post on 120 bugs in 120 days is particularly cool. 

Key Points

I think it’s best to start the post with some key points about Bug Bounties that probably aren’t clear or talked about by the companies themselves. However, they’re very relevant when you’re trying to decide whether this type of program is for you. Hopefully, this will provide a behind-the-scenes view of how ‘the crowd’ actually works.


  1. A huge proportion of the top bug bounty hunters are Penetration testers by day, who are employed by Security consultancies. According to Shubs, “a large amount of the top bug hunters on Bug Crowd and H1 are Penetration testers or security engineers for consultancies or in-house”. Looking closely at the leaderboards for other similar programs, it’s clear this is the case across the industry. Therefore, claims of better standards of testing or more technical expertise versus consultancies are generally unfounded. When you utilise a Bug Bounty company, you’re essentially calling upon a lot of the same resources who happen to use their downtime to find your bugs. The consultancies they work for also provide most of – if not all – the training for these guys too, so that’s where the expertise comes from. 
  2. A typical bounty approach does not cover a full pen testing methodology. I understand that some Bug Bounty companies have private bounties or additional programs that offer ‘full methodology testing’ and in honesty I haven’t had first-hand experience of the outputs. However, this sounds suspiciously like a standard bug bounty (with arguable more successful testers included by ‘invite only’) and penetration testing respectively. I’m sure it costs a lot more though. As the general premise of a bounty program is to only pay out on findings, this directly affects the behavior and approach of their testers. The upshot of this is that bug hunters tend to target critical and high risk issues (as they pay the most) and if they don’t find them within an arbitrary time window, they move on to the next bounty. Time is money for bug hunters and there’s other sites and other things to do at evenings and weekends. This means that coverage is not a priority when compared to a typical Penetration testing approach. Often, in a Pen test, lower risk issues can be chained to produce a higher risk finding – testers also frequently work in pairs and combine results. When you consider a crowd sourced approach where the players are working in isolation against each other, there isn’t the possibility of combining vulnerabilities as the other hunters cannot see all the issues raised while the program is running – nobody is tracking the big picture.
  3. You’re competing with other firms for tester time. Shubs made a really good point on this topic when I asked him about his methodology, he said “I pick out my bug bounty programs like a company would pick their ideal clients”. This is a really interesting point, as there is a lot of kudos associated within the Bug Bounty community for finding bugs in certain sites. This means that essentially, you end up with different tiers of clients in open bounties, with big name clients (such as Google, Apple, Facebook) receiving a lot more attention than lesser names. The exception being, when you pay more for a private program. The impact of this is that you compete with other organisations using a bounty site, often in what’s essentially a bidding war to get more tester attention. When you compare this with penetration testing at a consultancy, you don’t have the concept of trying to outbid other clients to get more time or better testing for your project.
  4. Bounty hunters are often specialist ‘farmers’ in a small number of unique and nuance attacks. From chatting with various bug hunters, one of the key trends I’ve noticed from the top guys, is that they normally have a handful of unique attack or recon vectors. These vectors are often wholly unique or a spin on known techniques. This means that top hunters will often try their own techniques across sites sequentially and move on quickly if they don’t have any luck. They’re unlikely to perform the typical methodology steps that are characteristic of a penetration test, again meaning that coverage has a low level of assurance. This means that as a client, you can often find that more common manually exploitable issues (that are non-trivial) can be overlooked as the top guys will move on quickly if they don’t get a pay-off within a reasonable timeframe.
  5. Criminals can hide in the pack and sell your vulnerabilities to someone else, rather than you. Quite often, the price that can be fetched on the Black-market for a vulnerability in a popular website can be an order of magnitude higher than is offered by a bounty program. This can lead to the temptation for bug hunters to turn to the dark side. This is one of the downsides to opening up your site to a bug bounty program, as it gives a plausible deniability to a BlackHat bug hunter and invites them in the front door with access and a ‘good reason’ to be poking around. Moreover, sites such as Zerodium legally buy and sell vulnerabilities in applications to the highest bidder. There’s an interesting conversation on this topic to be had around Wassenaar and copyright laws such as DMCA in the U.S., but that’s out-of-scope for this post. Beers sometime?
  6. Bounties often give clients what they want, rather than what they need. It’s not uncommon to see restrictions for certain types of bugs on a bounty. Quite often this covers some pretty serious issues, and even the top five of the OWASP top ten. This is obviously the client’s prerogative to run their program as they see fit. However, it’s a really bad security practice and one that you never see any self-respecting pen testing company adhere to. The reason I believe this is an issue, is because it shows risk acceptance of the most commonly found and exploited vulnerabilities on the Web. It’s like saying “I accept we have XSS on our site and I don’t care”. As simple and ubiquitous as XSS is, or indeed many other common issues are, they still represent a big risk to the end user and leaving these out of bounties perpetuates poor security hygiene at the most basic level.
  7. It can get noisy. If an organisation chooses not to select the option to have the Bug Bounty company perform validation on their bugs before they’re recorded, they’re in for a wild ride. From speaking with a large number of organisations who’ve run their own programs or not chosen this option from a third-party, it’s clear that the skill level of bounty hunters is extremely inconsistent. Again, this is an issue with the ‘anyone can join’ model, where many of the hunters blindly run intrusive scanners across your environment in the hope that their cracked version of Acunetix picks up something juicy. Compared with professional services firms where testers are interviewed, background checked and provided with indicative methodologies (as well as indemnities!), it seems like a false economy to take this approach. One notable exception (pointed it to me by @n0x00) is SynAck, who do actually perform background checks and offer training to their hunters, kudos to them.
  8. It won’t help you pass compliance for a penetration testing requirement. A typical crowd sourced bug bounty program will not satisfy compliance. I’m sure some of them will tell you differently, but the only way they’re telling the truth is if they’re doing penetration testing (or the auditor is easily fooled). Let’s take PCI 3.2 as an example. Requirement 11.3 contains a requirement to test applications (obviously bounties don’t deal with Network / Infrastructure testing) within it. It states that a methodology must be applied that follows certain criteria, most importantly for this example, the methodology “is based on industry-accepted penetration testing approaches (for example, NIST SP800-115)”. This is the first place Bug Bounties fall down, as they cannot provide a methodology, as each tester uses their own and I doubt any of them have created theirs based on the 80 pages of joy known as NIST SP800-115, the 213 pages of OSTMM 3 or any other legitimate standard. Mostly because, life’s too short. Secondly, the standard also states that all elements of requirement 6.5 must be covered during application testing, which requires some sort of statement attesting that this has been the case i.e. a de facto methodology. 
What Bounties Are Good For

  1. Managing a bug bounty program for your business. I would say this should be the primary focus of Bug Bounty companies. They do a fantastic job of finding nuance bugs from commonly studied applications. As previously mentioned, there are a huge amount of talented people who participate in this community already.
  2. Recon and automation techniques. Bug Bounties have really helped push the boundaries of automation and discovery techniques within the field of ethical hacking. Shubs has been one of the major proponents of this along with Jason Haddix from Bug Crowd, so huge kudos is due to them (and various others) for this. HackerOne also does a large amount of knowledge sharing publicly, which has contributed to the education of people at all levels of the industry.
  3. Keeping grey hats from going to the dark side (sometimes). Bounty programs have made security testing accessible globally and made it a career option for some people (especially in LEDCs), keeping them on the right side of the law.
  4. Providing experience and opportunities for newcomers. One thing that’s been apparent to me from interviewing candidates and also socialising with the community, is that bug hunting is a great route into the industry. Shubham told me about his back story in the process of writing this post, how he went from flippin’ burgers to having two InfoSec jobs (at a consultancy and his bounty exploits) and seeking (making!) his fortune. Both Shubs and Yappare have explained to me that they owe a lot to bug bounty programs, as do a lot of others who’ve been given a route into the InfoSec world.
  5. Marketing.

In summary, it’s my view that Bug Bounty businesses are trying to diversify into an area that do not fit their operational model. I believe that many of them are misleading clients as to the efficacy of their services and purporting to replace penetration testing and other security assessment services. I do believe there is a place for Bug Bounty programs in the InfoSec world, providing additional assurance to mature application owners who already do SDLC and Penetration testing as part of their security validation program (emphasis on validation!). However, they do not function well as a replacement for traditional penetration testing, as it’s not possible to assure (or in good conscience proclaim) proper coverage of the audit elements required for a good application security test. This cannot be built into a crowd sourced approach without first validating testing skill, methodology and most importantly motivation. In short, Bug Bounties are a ‘nice-to-have’ for those who have the money to include this approach as a security overlay.

I hope this has been a useful post, especially for those who’re considering procuring application security assessment services. I’m happy to respond to all (sensible) comments or questions, please add them below if you wish.

No comments:

Post a Comment