Tuesday, 31 October 2017

What Finnish School Children Can Teach The InfoSec Community

One of the plagues (in my humble opinion) within the InfoSec Community is compartmentalised thinking. We put different areas of our organisations into boxes and turn paradigms into silos. The artificial constructs, certification syllabi and subscriptions to schools of thought we use, structure our thinking and limiting our learning and interactions. This is not to say that any [particular] structured learning approach is wrong, but that we need to factor in the impact of these constructs on our thinking, learning and development as practitioners.
Alternative approaches to schooling have always fascinated me and made me somewhat envious of the beneficiaries. Consistently, research points towards the current approach to education as antiquated and at odds with the goals of inspiring learning and effective reasoning. It’s my belief that it’s never too late to fundamentally change your outlook and I’m consistently inspired by the innovative implementations of countries like Finland (and indeed, people I know in industry) to change the way I think, communicate and impart knowledge. This post talks about some of the different approaches to teaching / learning and how a joined-up approach can break damaging compartmentalised thinking.
Phenomenon-based Learning
The first thing I want to discuss is phenomenon-based learning. This approach was popularised by the Finnish schooling system in their National Curriculum Reforms in August 2016. The focus of ‘FBL’ is to improve the ‘authenticity’ of the learning experience, linking ideas to real-world scenarios and working in an interdisciplinary fashion.
They theory behind this approach is termed ‘constructivism’. In constructivism theory “learners are seen as active knowledge builders and information is seen as being constructed as a result of problem-solving, constructed out of ‘little pieces’ into a whole that suits the situation in which it is used at the time.”[1]. The approach also focuses on social interactions as a conduit for learning, drawing inspiration for how our understanding of language is developed. Essentially, the idea is to replicate how we learn more naturally as humans by not pigeonholing subjects and concepts.
In Finnish schools, this doesn’t totally replace taught subjects, but it augments some wider ideas and links all sorts of different facets (the common example given is the European Union) to real-world ideas. More information on the specifics of the approach can be found here.
How Are We Getting It Wrong?
Similar to the way in which school subjects neatly put ideas in pigeonholes, Information Security creates paradigms, career paths and exam tracks that shape thinking. This can create narrow fields of view and subscription to pre-conceived ideas related to a particular approach. Put simply, we’re creating worker-bees in an industry that needs to be creative and collaborative, as well as regimented. I’m not saying that any and all frameworks are damaging or have no value, but we should not allow them to narrow our sights. I see these types of structures as useful references to help guide us to make the right decisions, rather than a messianic answer to security. As an example, Agile development is something that has crept into this category and become a lifestyle brand. I think the more niche the area, the higher the chances are that it will become a cult-like framework and subsume a portion of practitioners.
While security audit is a useful tool, and frankly the only reason why a lot of organisations bother doing any security at all, it can create the idea that security and compliance are the same thing. This is destructive thinking and also creates a lot of the issues we see whilst going about our daily business.
In my view, few approaches are as effective in exposing the lack of joined-up thinking and processes than a live fire exercise (such as Red Teaming and attack simulations). Over the years, as I’ve orchestrated and run an array of attack simulation campaigns for organisations of various sizes, a few things have become clear about the players in the game (although not all, it should be said). I use the pronoun ‘we’ as I think we’re all guilty of this sometimes.
  • We don’t talk to each other, especially outside of our business functions.
  • We don’t like to concern ourselves with the roles of people outside our business unit.
  • We label ourselves and define our intellectual sphere based on our job function.
  • We measure ourselves against an arbitrary standard (often certification paths).
  • We think it’s someone else’s job to join the dots and think about the bigger picture.
  • We think it’s not possible to know everything, so it’s better to specialise and know one thing well, sometimes avoiding learning on this basis.

I believe that this type of attitude / thinking is something we learn early on and is not something easily undone. I’d be so bold to say that those who move outside this mind-set differentiate themselves, as people who’re useful in the security field vs. people who’re not. Moreover, the fundamental problem we have is the lack of quality thinkers who’re capable of creating holistic models and able to solve challenges collaboratively and creatively. I don’t think this issue is unique to security, but it’s certainly prolific and we’ve become a field that has more than its fair share of ill-suited people making up the numbers due to the skills gap. This probably sounds exceptionally harsh, but please let me explain. You could argue that this is real-life and there’s always a bell curve within a group of people. However, is this a trend that’s mirrored in other high-end vocational jobs, such as Doctors, Neuroscientists or Chemical Engineers? Is our capability as an industry proportional to the stakes and do we offer good ROI? It’s something to ponder.
The Future is Purple (and other Polychromes)
In scientific research, an interdisciplinary approach has been commonplace since the early 90’s. For example, Physics and Computing are combining to create quantum computers and genetics and neuroscience are unravelling the functionality of the brain. Should a joined-up approach be reserved for only grand scientific challenges? I don’t believe so. Setting this in the context of an organisation that is a high value target, under attack from a capable adversary, we can discuss the changes in approach and how more authentic thought processes can assist.
Phenomena-based adoption can help us re-evaluate our role in the bigger picture and help the coordinators conceptualise things in a more realistic way. We need to approach our challenges holistically and without paradigm bias. We should ask ourselves questions like: how do we know we accurately understand the real threats? If I’m a SOC analyst, do I know what a threat actor is and their MO? If I’m in the DevOps team, should I care about security outside of Availability? If I configure the Internet boarder technologies, should I care about compromises in user LANs? How does my role fit with the rest of the organisation?
Additionally, we need to consider what information we are getting following an attack and are we able to process or even understand it. Often the answer is that it’s overwhelming, and we need outside help. In this case, the most effective approach is to utilise existing expertise and knowledge and combine forces with the ‘attackers’ to understand how they do what they do and how to detect and prevent it. In the security assessment world, this is often referred to as Purple teaming (from the Red Team vs. Blue Team colour mix). This approach gives great learning opportunities for the defenders, in the problem-based style of sociocultural learning. There is also a reciprocal element as well, as the attacking team also gain a greater understanding of how difficult it is to defend. Typically, this doesn’t happen and it highlights a large issue currently in the professional services world, which is the lack of value given by professional services firms post-engagement. Collaborative phenomena-based learning is the future and purple teaming (or whatever you wish to call the idea) is key in our industry to better prepare to be more secure.
So, What Can Finnish School Children Teach Us as A Community?
I’m a big fan of bullet points and exclamation marks (and interrobangs)…
  • Break out of silos and compartmentalised concepts!
  • Don’t let curricula limit your learning!
  • Think critically about your role and remit within your organisation.
  • Seek learning opportunities outside of your role.
  • Security is an overlay of IT functions and human processes, therefore, it’s an advanced discipline and should be treated as such.
  • Security is everybody’s responsibility!
  • Learn in a style that is more authentic, read around topics.

Thursday, 13 October 2016

Bounties Bug Me (A Little)

The popularity of Bug Bounties has grown hugely over the last 3-5 years, with start-ups taking advantage of eager VCs and Angel investors who want a piece of the ‘Cyber’ pie. As this niche market has become more mature, growth opportunities for crowd sourced frameworks have peaked and pressures from investors to diversify have amplified. As a result, we now see emboldened assertions from some providers that their programs replace services such as penetration testing. For me, this is quite a stretch and I’m going to discuss the reasons why in this post, with some input from some of the world’s top bounty hunters.

I’d like to state from the off, that this isn’t an attack on Bug Bounties as a notion, but a counter to marketing by organisations in this space that misrepresent what these types of programs can logically provide. 

As a Senior Director at one of the world’s largest pen testing practices, I’m keen to stay as objective as possible, so I decided to reach out to a few prolific bug hunting friends to get their opinions and to keep me honest. Please note that my views are my own and not that of my employer. A huge thanks to Yappare, Shubham Shah and John ‘n0x00’ Carroll for their inputs. In order to give you some background of the quality of their inputs and why you should listen to these guys, I’ve included a quick overview of the contributors to this post. Yappare is currently (at time of writing) ranked 2nd All-time on BugCrowd and is the author of a fantastic blog, that can be found here. John Carroll is currently an AppSec gun-for-hire and one of the original Top 10 bug bounty hunters for Bugcrowd back in 2013, he was also one of the most prolific charity bounty hunters around. John’s blog is really well known for its humour and innovative techniques, check it out here.  Shubham Shah is regarded as one of the most innovative bug hunters in the world, regularly speaking at conferences about his tools and techniques to make the most out of ‘bounty time’, he’s also fairly prolific on HackerOne. Check out his blog here; his post on 120 bugs in 120 days is particularly cool. 

Key Points

I think it’s best to start the post with some key points about Bug Bounties that probably aren’t clear or talked about by the companies themselves. However, they’re very relevant when you’re trying to decide whether this type of program is for you. Hopefully, this will provide a behind-the-scenes view of how ‘the crowd’ actually works.

  1. A huge proportion of the top bug bounty hunters are Penetration testers by day, who are employed by Security consultancies. According to Shubs, “a large amount of the top bug hunters on Bug Crowd and H1 are Penetration testers or security engineers for consultancies or in-house”. Looking closely at the leaderboards for other similar programs, it’s clear this is the case across the industry. Therefore, claims of better standards of testing or more technical expertise versus consultancies are generally unfounded. When you utilise a Bug Bounty company, you’re essentially calling upon a lot of the same resources who happen to use their downtime to find your bugs. The consultancies they work for also provide most of – if not all – the training for these guys too, so that’s where the expertise comes from. 
  2. A typical bounty approach does not cover a full pen testing methodology. I understand that some Bug Bounty companies have private bounties or additional programs that offer ‘full methodology testing’ and in honesty I haven’t had first-hand experience of the outputs. However, this sounds suspiciously like a standard bug bounty (with arguable more successful testers included by ‘invite only’) and penetration testing respectively. I’m sure it costs a lot more though. As the general premise of a bounty program is to only pay out on findings, this directly affects the behavior and approach of their testers. The upshot of this is that bug hunters tend to target critical and high risk issues (as they pay the most) and if they don’t find them within an arbitrary time window, they move on to the next bounty. Time is money for bug hunters and there’s other sites and other things to do at evenings and weekends. This means that coverage is not a priority when compared to a typical Penetration testing approach. Often, in a Pen test, lower risk issues can be chained to produce a higher risk finding – testers also frequently work in pairs and combine results. When you consider a crowd sourced approach where the players are working in isolation against each other, there isn’t the possibility of combining vulnerabilities as the other hunters cannot see all the issues raised while the program is running – nobody is tracking the big picture.
  3. You’re competing with other firms for tester time. Shubs made a really good point on this topic when I asked him about his methodology, he said “I pick out my bug bounty programs like a company would pick their ideal clients”. This is a really interesting point, as there is a lot of kudos associated within the Bug Bounty community for finding bugs in certain sites. This means that essentially, you end up with different tiers of clients in open bounties, with big name clients (such as Google, Apple, Facebook) receiving a lot more attention than lesser names. The exception being, when you pay more for a private program. The impact of this is that you compete with other organisations using a bounty site, often in what’s essentially a bidding war to get more tester attention. When you compare this with penetration testing at a consultancy, you don’t have the concept of trying to outbid other clients to get more time or better testing for your project.
  4. Bounty hunters are often specialist ‘farmers’ in a small number of unique and nuance attacks. From chatting with various bug hunters, one of the key trends I’ve noticed from the top guys, is that they normally have a handful of unique attack or recon vectors. These vectors are often wholly unique or a spin on known techniques. This means that top hunters will often try their own techniques across sites sequentially and move on quickly if they don’t have any luck. They’re unlikely to perform the typical methodology steps that are characteristic of a penetration test, again meaning that coverage has a low level of assurance. This means that as a client, you can often find that more common manually exploitable issues (that are non-trivial) can be overlooked as the top guys will move on quickly if they don’t get a pay-off within a reasonable timeframe.
  5. Criminals can hide in the pack and sell your vulnerabilities to someone else, rather than you. Quite often, the price that can be fetched on the Black-market for a vulnerability in a popular website can be an order of magnitude higher than is offered by a bounty program. This can lead to the temptation for bug hunters to turn to the dark side. This is one of the downsides to opening up your site to a bug bounty program, as it gives a plausible deniability to a BlackHat bug hunter and invites them in the front door with access and a ‘good reason’ to be poking around. Moreover, sites such as Zerodium legally buy and sell vulnerabilities in applications to the highest bidder. There’s an interesting conversation on this topic to be had around Wassenaar and copyright laws such as DMCA in the U.S., but that’s out-of-scope for this post. Beers sometime?
  6. Bounties often give clients what they want, rather than what they need. It’s not uncommon to see restrictions for certain types of bugs on a bounty. Quite often this covers some pretty serious issues, and even the top five of the OWASP top ten. This is obviously the client’s prerogative to run their program as they see fit. However, it’s a really bad security practice and one that you never see any self-respecting pen testing company adhere to. The reason I believe this is an issue, is because it shows risk acceptance of the most commonly found and exploited vulnerabilities on the Web. It’s like saying “I accept we have XSS on our site and I don’t care”. As simple and ubiquitous as XSS is, or indeed many other common issues are, they still represent a big risk to the end user and leaving these out of bounties perpetuates poor security hygiene at the most basic level.
  7. It can get noisy. If an organisation chooses not to select the option to have the Bug Bounty company perform validation on their bugs before they’re recorded, they’re in for a wild ride. From speaking with a large number of organisations who’ve run their own programs or not chosen this option from a third-party, it’s clear that the skill level of bounty hunters is extremely inconsistent. Again, this is an issue with the ‘anyone can join’ model, where many of the hunters blindly run intrusive scanners across your environment in the hope that their cracked version of Acunetix picks up something juicy. Compared with professional services firms where testers are interviewed, background checked and provided with indicative methodologies (as well as indemnities!), it seems like a false economy to take this approach. One notable exception (pointed it to me by @n0x00) is SynAck, who do actually perform background checks and offer training to their hunters, kudos to them.
  8. It won’t help you pass compliance for a penetration testing requirement. A typical crowd sourced bug bounty program will not satisfy compliance. I’m sure some of them will tell you differently, but the only way they’re telling the truth is if they’re doing penetration testing (or the auditor is easily fooled). Let’s take PCI 3.2 as an example. Requirement 11.3 contains a requirement to test applications (obviously bounties don’t deal with Network / Infrastructure testing) within it. It states that a methodology must be applied that follows certain criteria, most importantly for this example, the methodology “is based on industry-accepted penetration testing approaches (for example, NIST SP800-115)”. This is the first place Bug Bounties fall down, as they cannot provide a methodology, as each tester uses their own and I doubt any of them have created theirs based on the 80 pages of joy known as NIST SP800-115, the 213 pages of OSTMM 3 or any other legitimate standard. Mostly because, life’s too short. Secondly, the standard also states that all elements of requirement 6.5 must be covered during application testing, which requires some sort of statement attesting that this has been the case i.e. a de facto methodology. 
What Bounties Are Good For

  1. Managing a bug bounty program for your business. I would say this should be the primary focus of Bug Bounty companies. They do a fantastic job of finding nuance bugs from commonly studied applications. As previously mentioned, there are a huge amount of talented people who participate in this community already.
  2. Recon and automation techniques. Bug Bounties have really helped push the boundaries of automation and discovery techniques within the field of ethical hacking. Shubs has been one of the major proponents of this along with Jason Haddix from Bug Crowd, so huge kudos is due to them (and various others) for this. HackerOne also does a large amount of knowledge sharing publicly, which has contributed to the education of people at all levels of the industry.
  3. Keeping grey hats from going to the dark side (sometimes). Bounty programs have made security testing accessible globally and made it a career option for some people (especially in LEDCs), keeping them on the right side of the law.
  4. Providing experience and opportunities for newcomers. One thing that’s been apparent to me from interviewing candidates and also socialising with the community, is that bug hunting is a great route into the industry. Shubham told me about his back story in the process of writing this post, how he went from flippin’ burgers to having two InfoSec jobs (at a consultancy and his bounty exploits) and seeking (making!) his fortune. Both Shubs and Yappare have explained to me that they owe a lot to bug bounty programs, as do a lot of others who’ve been given a route into the InfoSec world.
  5. Marketing.

In summary, it’s my view that Bug Bounty businesses are trying to diversify into an area that do not fit their operational model. I believe that many of them are misleading clients as to the efficacy of their services and purporting to replace penetration testing and other security assessment services. I do believe there is a place for Bug Bounty programs in the InfoSec world, providing additional assurance to mature application owners who already do SDLC and Penetration testing as part of their security validation program (emphasis on validation!). However, they do not function well as a replacement for traditional penetration testing, as it’s not possible to assure (or in good conscience proclaim) proper coverage of the audit elements required for a good application security test. This cannot be built into a crowd sourced approach without first validating testing skill, methodology and most importantly motivation. In short, Bug Bounties are a ‘nice-to-have’ for those who have the money to include this approach as a security overlay.

I hope this has been a useful post, especially for those who’re considering procuring application security assessment services. I’m happy to respond to all (sensible) comments or questions, please add them below if you wish.

Monday, 29 August 2016

When Two Worlds Collide: Why InfoSec Professionals Hate Recruiters

In honesty, I’ve never been overly fond of recruiters, stemming from my early days in the industry being duped into long journeys for interviews that were totally inappropriate, so the recruiter could make their number. However, it’s obviously unfair to tar all recruiters with the same brush, they’re just doing their job (to varying ethical and moral levels). I’m starting to see more and more posts from recruiters on LinkedIn, showing frustration at rude or terse reactions from the InfoSec community (especially Pen testers). I decided to write a post on the subject to discuss some of my thoughts on the topic and outline some of the key points on both sides, having used recruiters in my own career and also been on the hiring manager side.

What candidates need to remember is that recruiters are salespeople, they just sell people. What recruiters need to remember, is that they’re selling people and lifestyles, not things. A job is something that most of us spend a huge part of our lives doing and is therefore closely aligned to satisfying our various needs and wants. If you took the rather grim view of your life as a commodity, where you could buy and sell the hours within it, would you entrust a proportion of that to a middle-man (or woman) who is quite obviously dishonest, unethical and lazy?

Where Recruiters Go Wrong

As I start writing this section, I’m already pretty sure this is going to be the bulk of this post. I’ve experienced most of these gripes; a few on a daily basis. Some items on this list I’ve seen as a candidate and some are as a hiring manager – I don’t believe my experience is unique. I’m often pretty blunt with people who manage to reach me in these ways, mostly because I don’t agree with the approach on an ethical basis. I don’t think I’m alone.
  • Inappropriate Job Suggestions
This is really common; I think most people in InfoSec get these 2-3 times per week (if not daily). I think the thing that frustrates me most, is that it’s so obviously lazy. Our industry is obviously very security and tech savvy, we know the recruiter has done a LinkedIn keyword search and we’re one of the lucky people that popped up. Thusly follows a boiler plate email asking for a call about a role that is in no way close to what we do. The expectant chaser emails (also boiler plate) are always a nice touch.
  • Calling People in their office, often via generic or switchboard numbers
This is totally unprofessional and often leads to trouble for the candidate the recruiter is targeting – a few years ago a recruiter actually came to my desk phone via my boss, posing as a candidate, he thought he was pretty smart when he revealed his Ocean’s Eleven-esque scheme to me.
  • Sending CVs before getting permission
Again, this is totally unprofessional, not to mention in breach of the Data Protection Act. I’d recommend that if anyone discovers this is the case, that they report this as a serious matter to the agency concerned and possibly to the ICO. Moreover, part of me feels that individuals should be more cautious about who they send their CV to also. The recruiter will often try “I can’t tell you who the company is until you send me your CV”, this strict stipulation doesn’t often last long after you say you’re not interested in that case and then they spill the beans. The amusing part is that as the industry is so small, you usually know the key contact at the company and drop them a note to advise what’s happened – I’ve done this with candidates in some cases where their CV has hit my inbox in a suspicious or unsolicited fashion. I know many others who kindly do the same.
  • Cut and Paste exploratory emails
As a hiring manager I get these pretty frequently. A cut and pasted boiler plate introduction on LinkedIn about how successful the recruiter has been and how I should consider using them. They obviously really want my business and to build a relationship THAT much, they took the time to change the name at the top. If I tried to work with clients at Director level and above with this approach, I’d not last long. We’re not short of networking events, conferences etc. In my view there’s no excuse for the junk and scattergun approach.
  • Pretending they’ve spoken to you before or know you
This drives me nuts. Normally, the email (or InMail) starts “I just tried to call you” or “We spoke some time ago”, neither of which is true. It’s often a subtle reference, but for some reason some recruiters think you won’t realise. It’s such an obvious trick and for pen testers (who spend their whole day trying to think like a sneaky criminal) we spot this a mile off. Let’s at least try and start with honesty.
  • Pretending they know someone you do
Another really silly and dishonest mistake is thinking that people don’t talk to their friends. If someone says “I got your name from Ben, we’ve worked together and he told me to call you”, the first thing I’m going to do is send a message to Ben along the lines of “wtf” or “orly”. Lies and deceit are not the way to form long lasting business relationships.
  • The recruiter explains the industry to you
It’s always great to have someone with a couple of years’ experience in recruitment tell you how your job works and what people are looking for. General coaching is really useful when you’re quite junior, but a lot of people find this strange and somewhat embarrassing.
  • The recruiters job title is “Information Security Consultant” or similarly misleading
I’ve seen a few threads online where people complain about this, I tend to agree that this is a poor practice and is largely perpetuated to trick people into connecting with a recruiter, thinking they are a peer within the industry. It can be quickly remedied by viewing the person’s profile before connecting, but that doesn’t negate the dishonesty.
  • General Shenanigans
<insert horror story or anecdote>

The Other Side

Not all recruiters are bad. I’d say the good ones are certainly in the minority from my experience, but we should remember that they’re people who’re just trying to make a living too. The reality is, that recruiters aren’t going anywhere anytime soon, so we should grow thicker skins and work on solutions, rather than complaining all the time. Companies are always going to use recruiters to find talent, whether it’s to obscure their hiring activity from the public eye, save money or they just like to troll us. We can either help improve the situation or watch as it gets worse. There is actually an ombudsman in the UK for recruiting firms, called REC (https://www.rec.uk.com/membership/compliance/code-of-practice), but it requires agencies to join before they’re subject to the rules and it’s not widely mandated. I’ve also seen a recent effort from CERIS (https://www.getfeedback.com/r/1V16FODf) to create an industry body dedicated to recruitment practices within InfoSec, however, they are yet to get a proper website and talks of alignment with industry bodies such as CREST appear to have fallen down.

Room for Recruiters in InfoSec?

In smaller niches of the industry, such as penetration testing, it’s easy to think that everybody knows everybody else – so why do we need recruiters, right? Yet out of the ~1500 testers there are floating around, probably half are vaguely good (I’m being generous), then maybe a fifth of those may be open to a move at any one time. Then, there are multiple levels of seniority – and that’s only counting experienced hires. So, how do you find the good ones? Most organisations don’t have the resources or relevant skills (in their HR dept.) to search for these types of people and anecdotally speaking, they rely on  internal recommendations. As a Director (and hiring manager) of a rapidly growing large consultancy, I’d put recruiting as one of the top two priorities for me (probably only second to culture). I probably attend more than ten conferences per year (technical ones such as 44con, Kiwicon, ZeroNights, Brucon, Ruxcon, HITB, DerbyCon, BlackHat, Defcon) with one of the key goals being finding talent. I’d say that this (combined with trawling blogs / github) gets me about 75% of the way there and costs me a lot of time and my travel budget. The other 25% takes me hours of trawling LinkedIn (I don’t use external recruiters right now). For smaller organisations or more corporate ones, this simply isn’t an option, and I can see how the typical ‘pay for results’ model is appealing.

I feel there is a place for recruiters in our industry, but for nowhere near as many as there are now and they should try a lot harder to understand the different disciplines, qualifications and experience levels as well as exercising a basic level of ethics and honesty. However, I feel a lot more could be done by industry bodies to assist in this area, providing job boards and independent mechanisms for candidates to find roles.

Friday, 27 March 2015

Penetration Testing: You’re Doing it Wrong (?) – Part One

Sexual innuendos aside, I've wanted to write an article about the unspoken thoughts of penetration testers (at least my own and the great testers I've been lucky enough to work with) for quite some time, but 100 hour weeks and international travel for work tend to get in the way! The main focus of this post is to describe the typical approaches of both the security assessment services industry and organisations that consume them. I have given particular focus to new attempts to make testing more realistic and how this compares to what I term as ‘Traditional Penetration Testing’. I also touch on actionable OSINT and Threat Intelligence, as well as Threat and Risk modelling as a function of assessments strategies. The context of this post assumes a medium to large sized organisation and at least a moderate level of maturity such as maintaining an accreditation framework such as PCI DSS or ISO27001.

What Do I Define as Traditional Penetration Testing?

I describe a traditional penetration test, in much the same way as industry and accreditation bodies do, which is in terms of application and infrastructure testing. Typically, a large organisation will have two main streams of testing that pinpoint the new and the old. Business as usual (BAU) testing, includes existing infrastructure and applications that are tested on an annual (or periodic) basis, which includes compliance driven assessments. Project-based testing often comprises wholly new applications and infrastructure that need to be assessed before going ‘live’. The two most common (in my experience) approaches to servicing this requirement are a penetration testing framework that involves multiple suppliers (often using a round-robin approach to avoid claims of inequality) and a single supplier approach. Additionally, it’s worth noting that SMEs will typically use a single or multiple suppliers to service ad hoc requirements and will often create RFPs for each requirement.

So, Why is This Wrong?

One of the key criticisms of this approach is that it’s unrealistic. For example, performing an eight-day black box Web application penetration test in your staging environment, is the equivalent of building a temporary six-foot wall around your brand new house and attesting the security of the en-suite bathroom by getting a fat kid to lean against the front gate for a couple of hours and play knock-down-ginger with their bespectacled best friend. This criticism is something that I agree with, as tests often lack context, realism and their approach is based on improper calculation of the likelihood of attack and compromise. Often, this type of approach is justified due to the low business impact or revenue generation of the asset or cries of ‘industry best practice’. The truth is that baked-in ‘security as a feature’ - with its genesis in SDLC, secure architecture and user education – consistently provides better return on investment (not to be mistaken for cheapness) and more robust attack surfaces than atomic assessment of network sub-sets and standalone applications.

Moreover, many key elements of how threat actors will approach an attack are left out, as vectors such as (D)DoS attacks and social engineering are ubiquitously omitted from scopes. There are obviously ‘good’ reasons why this may be the case, such as cost, perceived ROI, perceived risk and legal implications. Fundamentally, most of these assumptions are wrong. There are ways of reducing these types of risks to acceptable levels for almost all scenarios - so why don’t people do it? I think the most correct answer is that the decision / policy makers don’t know how, and more importantly they don’t want to admit they don’t know how. It’s a very safe option to subscribe to conventional (circa 1995) wisdom and take an additive approach to anything new, giving a hat-tip and a wink to the industry and your peers that you’re at the forefront. This often involves tacking ‘bleeding edge’ services or appliances on to end-of-year budget surplus rather than questioning the value of what’s being done and going down the rabbit hole bottom-up (ooo err), armed with the detail of new approaches.

Another key cogitation, is the quality of testing and the ultimate understanding that’s passed on to those who design and create IT systems and infrastructures. As traditional penetration testing services are largely undifferentiated and a commodity (in the UK market), it’s difficult to know what ‘good’ is, even within the scope of a concept that is arguably broken. The industry is largely underpinned by the CESG CHECK scheme as a baseline for individual skill, with CTM and CTL status being used as metrics. In reality, the testing quality you get depends on the individual doing the testing, and not so much the company you hire (although this affects the overall experience as a consumer). As a minimum, most pen testing service providers will normally quote a framework or a baseline that they align to (such as OSTMM, PTES or NIST 800-155), have a defined methodology, and in lots of cases a checklist they follow when testing. A good example of a baseline is the OWASP top 10, used as a measure for assessing web applications. The OWASP top 10 is simply a list of the top 10 most common vulnerability categories discovered on the Internet by OWASP. The issue with these types of measures, is that your top 10, may not be the OWASP top 10 and you may be missing key tests due to lack of context and a one-size-fits all approach. From experience of testing frameworks, if a client gives a supplier 100 web applications to test over the course of the year it’s likely that most of the vulnerabilities discovered will be repeated again and again due to reuse of badly written libraries or learned mistakes during the development process and server build / configuration. How many paid-for hours could be negated by trending analysis, code review of re-used libraries, documentation of secure builds and hardening rather than testing what’s essentially the same application over and over again and finding the same issues. It’s not uncommon to see this carry over year on year, with a new testing firms being brought in to validate findings.


To conclude, I believe that a lot of the shortfalls in basic security hygiene come from the people running the show (read CISO, CTO, InfoSec Manager). There is a simple lack of understanding of Cyber threat / risk and how to quantify, prioritise and remediate. It’s a lot easier to not rock the boat, make a metaphorical pinkie swear with China and North Korea to the effect of ‘don’t ask don’t tell’, than to admit you don’t understand your attack surface or how to manage it.

And then?

I think that I've carried on quite enough for a single post about the issues, so I’ll be continuing in a new post on how I believe these issues can be remediated and a detailed discussion of: simulated attacks, CREST STAR and CBEST, the pitfalls of changing tact, and the risk of doing nothing.

Monday, 26 May 2014

What You Need To Know to Become a Penetration Tester

It really has been a long time since I last posted. This post is more of an essay, so it may be a TL;DR for some, but hopefully a there is some good information for those who wish to break into Penetration testing or at the very least something I can point people to next time I'm, asked.

As I’m sure is the experience of other Penetration Testers, I’m often asked (or see slapped across LinkedIn Forums) by a whole range of people “How do I break into Penetration testing?” or the like. The prospect of becoming a ‘professional hacker’ is all too enticing for graduates, IT professionals and even Information Security bods in other functional areas alike. Having answered this question and posed many a question in rebuttal, I decided to formalise my experiences, musings and advice into a single blog post. I hope it helps.

What is Penetration Testing?

To begin with, it’s always worth checking your understanding of what a Penetration tester actually is, does and some of the attributes belonging to the ‘good and great ones’. Being a Penetration tester is not the same as being (what the media often portray as) a hacker. To dispel a couple of myths about Penetration testing, it’s worth describing some of the aspects of the job in bullet points for brevity.
  • For most of the industry, a Penetration test does not involve development of 0-day exploits.
  • Normally, you get a very limited amount of time (days) to compromise systems that hackers would have months to do.
  • You always have to document your findings in a written report at the end of the test. Often, you will be asked to explain in detail some or all the findings to technical and non-technical audiences.
  • In a lot of cases, clients aren’t looking for a single route to domain admin. They’re interested in their broader attack surface and coverage across their whole estate. 
  • Business considerations, such as loss of earnings due to downtime and cost of engagement are much more important than you may initially think (or probably not consider at all before testing).

What makes a good tester?

I believe that there are quite a few considerations beyond raw technical knowledge that make a good tester. It’s key to remember again that Penetration testing is not ‘hacking’ and although there is a place for the borderline-autistic who hacks the Pentagon on their neighbours’ wireless, it’s more likely they’ll be suited to research or the Russian mafia. Again, I’ve added a bullet pointed list to describe some of what I consider key attributes of good and great testers that I consider when I hire non-experienced testers.

  • The right attitude – This is much more important than someone who’s done a course (CEH or the like). The truth is, I don’t believe that you can be a good Penetration tester if you’re not passionate about IT Security and moreover technology. It does need to be more than a job, as the up-skilling and constant personal development is not possible to maintain if your heart’s not in it. Also, you need to be fairly autodidactic (into self-learning) as even now there are few good core texts on most topics and they become outdated quickly. Most experienced tester will have gone through a huge amount of self-learning and will be resentful if you expect it on a plate (others just plain don’t want to share).
  • Good fundamentals – The best testers know a lot about a few things, but something about everything else. Despite some academiphobia (is that a word?) within the industry, I think that an IT related degree provides a great grounding in computer science and will stand you in good stead. Similarly, experienced sysadmins, network architects and developers should have good foundations on which to build. It’s common to find testers with big knowledge gaps, it’s really important to understand all areas of enterprise infrastructure to some degree in order to progress through the ranks.
  • Technical Prowess – At its core, Penetration testing is an extremely technical discipline. Not only do you need to understand how things work at a low level, you need to subvert controls in a repeatable way and learn constantly as new versions of software/hardware are released. The ability to code or script is always an advantage, even if you’re limited to simple bash scripting. However, some of the best testers I know can’t write code but can read and break very well indeed. That said, it will limit your vision and scope ultimately if you can’t. More on this later!
  • Soft and Written skills – Often overlooked, these skills are what separates Penetration testers from hackers and script kiddies. Ask yourself, would you as a client accept the work of someone who cannot write a coherent sentence, who cannot express simple issues in plain English and pay over £1000 per day for the pleasure? The key deliverable for the client is the written Report. Penetration testing companies require consultants who can read, write and speak English well. Unless you’re a total genius who’s finding 0-days nobody else can, they’re unlikely to overlook total ineptitude in this area.

Where to Start

In general, I would say that it’s almost impossible to get into a Penetration testing job at any level if you lack any exposure to computer science or hacking. It’s not like sales where you can start at the bottom with nothing and there are thousands of jobs.  
From my experience, there are four common starting points that lead people to the Penetration testing path. Apologies if this doesn’t fit your circumstances or I’ve left out anything that’s obvious.
  1. A school or college leaver who hacks as a hobby.
  2. An existing IT professional (e.g. Sys admin, Developer, Network Engineer)
  3. A recent graduate of Computer Science, Cyber Security or Ethical Hacking
  4. A graduate or experienced professional in another field
Scenario 1

In Penetration testing, having a lack of schooling and higher / further education isn’t necessarily a hindrance if you have some skill. However, you will likely need to prove this to a greater extent than perhaps a graduate would. Bug bounties are a great way to prove your prowess, with sites such as Bugcrowd paying sizeable winnings to the best bug hunters. If this is beyond your current skill level, it’s worth playing with some of the teaching frameworks such as Metasploit Unleashed or DVWA to hone your skill. It’s worth remembering that companies pay large amounts of money to recruiters, so if you approach them directly with a well-written CV, they’re likely to appreciate your enterprising spirit as well as the potential cost savings! If your CV is short or particularly unimpressive, you should pay particular attention to writing a covering letter or email. This should be well constructed and demonstrate your keenness to work for that company specifically (everyone likes a bit of mild flattery) and a bit about your background and aspirations. It’s worth noting that most (me included) organisations will throw out emails or CVs that are poorly written or full of mistakes at first sift. If you’re not grammatically gifted, ask someone who is to assist.

Scenario 2

Existing IT professionals already have quite a bit of skill (potentially) in a useful area. This is often quite attractive for Penetration testing companies as they understand general working practices and professionalism (one would hope!) and are often very keen! My advice would be to avoid expensive courses and retraining in order to land a job, but to focus on moving into a role as quickly as possible. There are plenty of cheap or free resources online to gain some basic knowledge around testing to move up to a level whereby you can hold a decent conversation in an interview or demonstrate working knowledge of Metasploit or Burpsuite on a rig. I always believe that attitude is more important than current skill-level with inexperienced hires, and spending £3k on a SANS course won’t make you start any higher in an organisation without testing experience. Great resources for this type of up-skill can be found on sites like SecurityTube, Udemy, OWASP (or just YouTube) and courses like Metasploit Unleashed and Mutilidae will help you reach initial goals.

Scenario 3

As a recent graduate, you have likely been exposed to good basics across a wide breadth of IT; some of it you may even remember. For those of you who have done a Computer Science degree and particularly enjoyed a security module or completed a Security or hacking related dissertation, you may feel the next step is to apply for a Penetration testing job. In this scenario, my advice would be to make totally sure it’s what you want to do and commit. As an interviewer, I’m always looking for a passion for security and some evidence of learning outside of University in this field, or as a minimum, a good security-focused FYP. If you’re lacking this, it will likely seem as though you don’t know what you want to do and you saw a couple of jobs being advertised for senior Penetration testers for up to £100k. For the people who’ve done a Security-related degree (or Masters) at Universities such as Abertay, Glamorgan, Kingston, Royal Holloway etc. you’re likely to get to the interview stage based on the specificity of your experience (as long as you got a reasonable grade).

Scenario 4

If you’ve no experience or qualifications in the field, then it’s likely to be a struggle to get an interview on the strength of your CV alone. I would advise the same approach as a School or College leaver, get ethically hacking and learning! An impassioned covering letter and interest in being an unpaid intern will often turn the heads of a hiring manager.

Industry Accreditation

This is probably the most frequent area of questioning I get when people ask me about moving into the industry. As the industry is so heavily focused on the CESG CHECK scheme, it’s best to separate the discussion into two parts, the CHECK scheme and other qualifications.
The CHECK scheme was initially set-up in response to the increased demand for skilled Penetration testing to be performed against HMG (Her Majesty’s Government) assets and CNI (Critical National Infrastructure) in the form of ITHC (IT Health Checks). In short, the Government didn’t have the funding to hire enough heads internally to complete all the work at the right level (or the salaries to tempt good testers away from industry). Therefore, the CHECK scheme was created as a framework to manage the subcontracting of this extra effort. A large amount of Penetration testing firms rely heavily on work from the CHECK scheme and it remains a huge source of income. Both companies and individuals can accredit to the scheme, with approved organisations deemed as ‘Green light’ should they satisfy the required criteria. 

The accreditation of individuals is quite complex, and even experienced testers still sometimes get confused by the different options. There are essentially two grades of Penetration tester within the scheme, CTL (CHECK team leader) and CTM (CHECK team member) with the former being the top tier. In order to achieve either of these levels the individual must pass an exam, hold SC clearance and work for a CHECK green light company. SC clearance can be attained by UK and Foreign nationals’ resident within the UK. However, if you’re not a British citizen you will often be caveated with ‘UK eyes only’ on your clearance, which will stop you from doing certain types of work. At the time of writing, the following table shows the possible examination paths to accreditation.

CHECK Team Leader
CHECK Team Leader (Infrastructure)
CREST Infrastructure Certification Examination (www.crest-approved.org)
Tiger Scheme Senior Security Tester (www.tigerscheme.org)
Cyber Scheme Team Leader Examination (www.thecyberscheme.co.uk)
CHECK Team Leader (Web applications)
CREST Certified Web Application Tester (www.crest-approved.org)
Tiger Scheme Web Application Tester (www.tigerscheme.org)
CHECK Team Member
CHECK Team Member
CREST Registered Tester Examination (www.crest-approved.org)
Tiger Scheme Qualified Security Tester Examination (www.tigerscheme.org)
Cyber Scheme Team Member Examination (www.thecyberscheme.co.uk)
Fig.1 - Taken from CESG website

The main difference between a CTL and a CTM (apart from the difficulty of the exams) is that a CTL must author the report and lead the testing. A CTM cannot work solo on a CHECK test. Therefore, a CTL is more valuable to a company that performs CHECK work as they can work alone, which makes scheduling more easy and it’s likely the tester is more highly utilised. The CTL grade subdivides into Web Applications and Infrastructure, however, there is currently no restriction on what type of work a CTL can perform either on their own or with other testers (i.e. a Web Application CTL can perform Infrastructure work solo).

Most testers will aspire to pass these exams, certainly when working within a CHECK Green light organisation, as their employer will either mandate or encourage it. The CTL exams are all-day and quite challenging, even for experienced testers. More information on the specifics of the exams can be found on the sites listed in the table above.

Other Industry Accreditations

There are loads of courses and accreditations out there, I have just listed and given a few lines about the courses and my opinions on each in order to give a flavour of the common ones.

CEH – The general consensus of testers in my experience is that CEH lacks technical detail and isn’t really a very good qualification for a Penetration Tester. However, I have not completed this exam myself and base my opinions on the experiences of trusted colleagues and the course syllabus and examination guides. I wouldn’t recommend this certification if you wish to be a Penetration tester.

OSCP / PWK by Offensive Security – This is a pretty technical course, which many of my team and colleagues have done. The course uses Kali Linux as a base operating system and runs you through a challenging series of labs exploring scripting in bash and python, the basics of exploit development and loads more. It’s online and give you VPN access to the various labs required for the course. In order to gain the OSCP qualification, you are required to submit coursework from the labs and complete a challenging exam over the course of 24hrs. Feedback on this has always been that it’s awesome and very challenging.

SANS Courses – Again, I have never been on a SANS course, but I have seen a lot of the material (having put some of my team through these), know a few instructors and have heard good things across the board. My one criticism is that most of the exams are multiple choice and not practical-based, which is a bit disappointing. That said, the instructors of the on-site courses are always excellent and if you can keep up you’ll learn a lot. I believe that it is more of a standard in the US to go after the GWAPT or GPEN exams.

BEWARE recruiters

I decided not to do a section on CVs, as I felt it was a bit patronising. Suffice to say, don’t send your CV across with any mistakes (spelling, grammar or otherwise), don’t leave any unexplained gaps and don’t in clude huge sections on your non-security related hobbies.
I supposed that finding a job often starts with recruiters, or at least LinkedIn. I have a hard time working out which is which nowadays, but it’s where I shall begin. For a lot of companies, recruiters are a necessary evil, but it’s best to attempt to avoid going through one unless it’s a necessity (some larger companies may mandate this route).

I have literally hundreds of horror stories about recruiters in Penetration testing, which I won’t bore you with, but be wary of sending them your CV (and moreover them getting hold of your CV, sending it to loads of companies then ringing you to see if you’re interested in any that bite). I would advise that you take their advice with a pinch of salt as most don’t understand the market. You can waste a lot of time being sent to far-flung places for inappropriate jobs that a recruiter persuaded you was a good idea (in order to make their quote of candidates to interviews). If you’re unsure, don’t be afraid to ask to speak to the company on the phone first to clarify the role is suitable.

I would advise that you approach reputable companies directly, you can find a list of CHECK Green light companies here for reference. 

If you’re following my advice and taking a direct approach, I would always advise that you check the company website first to see if they mention hiring on there. As I mentioned earlier, most organisations will see it as a positive thing if you track down the right person (or at least an appropriate one) on LinkedIn, send them a bespoke email introducing yourself and asking them about hiring and if they’d consider you for an interview. This saves them money, and shows a bit of initiative on your part.

The Interview Process

Most interviews I’ve done (as an interviewee and interviewer) for testing roles have all followed a fairly formulaic approach. They usually follow a two or three stage process with a mix of the following aspects:
  • Phone-based – Some simple technical questions (Nmap flags, Port numbers, What’s a XSS? etc.) and to check you speak good English and have a brain.
  • Face-to-face (Technical) – This normally is a series of technical questions and a technical assessment on a rig of some kind, often with you explaining as you go or the interviewer(s) watching on another screen what you do. This can be very intimidating as they tend to bring in very senior technical people to watch. Normally this will be pretty basic like popping a box with MSF (using MS08-067) or basic XSS.
  • Face-to-face (Presenting / meet-the-director) – You may be asked to present a Penetration test report or prepare something to assess your interpersonal skills. Depending on the company, you may need to meet a higher level business representative such as a director or partner.
  • Face-to-face (General) – A general interview will have a mix of discovery questions, often using HR methodologies such as the STAR model. This type of interview will seek to find out a bit more about you as a person and your aspirations to check for a cultural fit.

Before you go to the interview, it’s always worth asking what will be required and if you need to prepare anything. Interviews are an opportunity to sell yourself, mention anything related to the field of penetration testing that you have done. I recall one specific interview with a very shy gentleman, he really struggled explaining to me some really basic concepts such as what port scanning was and had never heard of the OWASP top 10; things were taking a nose-dive. Suffice to say, he hadn’t done well and didn’t appear to have a clue about anything in security, after having quite a promising CV. At least until I asked him the question ‘what do you do in your spare time?’, to which he answered “oh, I like exploit development and malware reversing normally, you know that bug that was found in IE last week, that was me, I got £10k from Microsoft for that”. Totally mental that he didn’t mention that, even after I asked him the ubiquitous ‘tell me about yourself?’ at the start of the interview.

I hope that my post has provided a useful amount of information to aspiring Penetration testers, if you have any comments, criticisms or more questions please leave them below.