Standards for Penetration Testing

The cost and quality of penetration tests vary wildly between different vendors. As a response to those differences, a group of security professionals have been developing the Penetration Testing Execution Standard (PTES). We solicited some comments about this standard, and standards in general, from several people including:

• Christopher Nickerson of Lares Consulting and one of the founding collaborators of the PTES
• Pete Herzog managing director and co-founder of ISECOM which publishes the Open Source Security Testing Methodology Manual
• Tim Grance a computer scientist at National Institute of Standards and Technology
• Rob Havelt director of penetration testing at Trustwave's SpiderLabs

Christopher Nickerson

What is the purpose of the PTES?

PTES is a new standard designed to provide both businesses and security service providers with a common language and scope for performing penetration. The industry has used the term Penetration Test in a variety of ways in the past. This has driven a large amount of confusion to what a Penetration Test is or isn't. We aim to create a clear standard to measure Penetration Testing and provide customers/consultants a guideline to how testing needs to be conducted. This will create maximum value to the client and insure they get repeatable and measurable QUALITY services. All of us involved feel that the lack of definition has allowed for the term to be polluted and its value has been severely watered down. This is bad for both the Penetration Testing community and the companies providing the tests.

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Download Now

FREE role-guided training plans

Get 12 cybersecurity training plans — one for each of the most common roles requested by employers.

Download Now

The organization has been growing, how did this all start?

It has always been in the works. Most of the founders have been talking about the need for this standard for YEARS. In November 2010, I was fighting my way through a client's previous "penetration test" and it was the last straw. I have (as have so many of us) seen hundreds of "penetration test" reports that are nothing more than a scan with no other value provided. This report I was reviewing was in that same vein. In haste, I sent out an email to some of the members of the penetration testing community who have had the same complaint for years. In my email was a call to action and a plea to have them be part of the solution and finally put this problem to bed. Thankfully, every single one of them responded with a resounding "Yes." From that day, it has steadily grown to over 50 members, each with a decade or more of InfoSec/Penetration Testing under their belt.

Is there a timeline to achieve certain benchmarks on this project?

I don't want to jinx us. Let's plead the Fifth on that one. Well, in all honesty, we have a date in mind and will release the date shortly. For the questions sake, I will say ASAP! It is so needed, that we are busting our tails to get this out there and provide a reference to the large group of clients that have no guidance as to what they should be getting.

Aren't there already some government and open source standards for penetration testing?

The only one out there that attempts to thoroughly address it has been OSTMM and it is still addressing about 20 percent of what we discuss in our standard. The issue with the current standards is that they have been built with compliance in mind and not the business.Their type of standard allows for some protection to be tested around the "asset" of that compliance's focus (For example, PCI has "card holder data" as a focused asset). They do not evaluate the business and the real risks to their individual needs.

OSSTMM is good for a general security mindset but does not provide a specific reference of how/exactly what/ and example goals of each type of test.

And what about NIST Special Publication 800-115?

Section 5 provides some guidance but it is even more limited than OSSTMM. That SP covers some of the method/mindset and touches on an even smaller group of 'how' to execute. All of these standards use vague language and rarely have adequately defined goals to let the reader understand What/ Where/ Why / and HOW the execution will occur.

Think of what we are doing as an OWASP type document for pen testing.

Pete Herzog

We started the OSSTMM because of the number of penetration tests that were done with mysterious, proprietary methods and we wanted something public that the pen test service providers could use to make a more thorough test of operations and their clients could use to have a better idea of what was being done. Early versions of the OSSTMM attempted to define types of tests as well, like what is a penetration test. However we stopped that in OSSTMM 3 due to conflicts between marketers, security testers, other standards bodies, and the customers.

Older versions of the OSSTMM also provided a list of expected results for various test modules. However, as we grew to a methodology which can test the security of anything, especially advances for which no security testing methodologies previously existed, like cloud computing, high frequency microprocessor testing, and trusted computing, we found the Expected Results section to be vague and too limiting. We are still using it though in Applied OSSTMMs which are practical manuals that use a subset of OSSTMM modules and tasks for specific technologies and services.

In the meanwhile, security testing as a service has gotten better but for the most part it's still right up there with dentistry as a blind-trust service. In the dentist's chair you can't see what they see, you can't interpret what they analyze, and you feel like you need to take them at their word about the damage that is there or else they tell you that you will suffer some awful consequences in some vague future time. This is what the OSSTMM sets to improve upon by standardizing what to test so as to provide clear results and attack surface metrics which don't require the client to be a magician's assistant to know where the results came from.

And in general, how do you feel about a standard for something that requires creative problem solving like a penetration test? Is there a downside to having a standard? What is the upside for clients?

The OSSTMM is a methodology and as such it provides what, when, and where. We didn't cover the how because of the resistance we encountered in the industry at the time. The community wanted flexibility to work creatively and did not want to be held accountable for doing certain tests in certain ways as one is with a standard.

In pen testing specifically, where each engagement is as much a test of the pen tester's ability as the infrastructure, there is an enormous amount of flexibility required by the tester that really leans away from requiring how something should be done.

In the past, pen test companies felt that being told how would really restrict them too much in the deliverable they provide for their client. The other side of the coin is the danger that it invites new penetration testers to the industry who ONLY do it that way as the bare minimum, something that Top 10 and Top 20 list makers have run into as well.

When you consider that a pen test is a negotiation of vulnerable interactions across all operations and each organization has a unique operation, it becomes difficult to match a specific how-to for every type of operation. While this can be done very generically, you then have the problem of this becoming a way to placate the testers (did they do everything listed and still not get in then okay, it's secure and they couldn't have anyway) rather than finding the right tests for testing the infrastructure. However, if the industry has changed and this is what testers and clients want now, then it should certainly be addressed.

For those buying pen tests, the best thing about having a standardized way of how to do something is that it allows all service providers for this to compete on equal footing. That makes it much easier for the consumers to pick a penetration test provider because then the real difference is often just price. In an interconnected world, we already see tests bought and performed by penetration testers in certain countries like Spain, Mexico, Brazil, and Chile because they have great penetration testers too and are usually much cheaper. The onus is then on the pen test service providers to differentiate and avoid being a commodity.

That's very different from those penetration testing companies providing an OSSTMM security test which does not standardize how to do the testing of their unique operations but instead provides clients with a means to get standardized results and attack surface metrics. That allows them to compare with other OSSTMM reports from other penetration testing companies and even in-house testing. Since only the results are standardized, pen test services can compete on the thoroughness of the scope tested for the price and the customer still gets a report usable for tracking and comparing security changes.

Tim Grance

It appears the aims of the PTES are laudable and their goals of a more detailed and descriptive processes are reasonable. I look forward to seeing the finished product. We encourage their efforts. Using NIST guidelines such as 800-37, 800-53, and 800-115 with well-tested and vetted private sector resources is also encouraged. In doing testing, no guideline or process will be all-encompassing or far sighted enough that it envisions every situation or scenario. Sound judgment, deep knowledge and wide breadth of current and emerging threats and technology, experience, and high quality training leavened with common sense are needed to deliver high value actionable results.

Rob Havelt

I know many of the principals involved in creation of the outline and mind map - including folks from my own SpiderLabs organization. I've even reviewed/contributed some to it.

I think PTES might be a bit different in purpose from PCI or other standards like that. The driver for PTES is the fact that there is no standard defined term for a Penetration Test. However, the concept is one that is "sexy" and everyone wants to be able to deliver one — even organizations that lack apparent skill, talent, and general know how. But this does not stop them. They believe that running bulk vulnerability scanners and a few auto-exploitation tools in an environment somehow qualifies as a Penetration Test. They actually charge money to come in, bungle around for a few days and leave the client with nothing other than several thousand dollars less in their security budget that could have been spent on something useful. This is obviously an unacceptable situation and one that desperately needs some correction. I believe it was for this reason that the principals gathered together various offensive security professionals to build PTES.

Don't get me wrong - I am still very skeptical of standards. And if PTES ever resulted in a "standard" checklist methodology that everyone must follow, I'd be giving talks everywhere about why PTES based tests are useless will be a huge waste of your time and money. But I'm not sure that's the direction in which they are heading.

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Get Your Plan

What should you learn next?

From SOC Analyst to Secure Coder to Security Manager — our team of experts has 12 free training plans to help you hit your goals. Get your free copy now.

Get Your Plan

So yes - I'd be a person that would not be a big fan of a "standard" pen test. But I'm also a huge enemy of "tests" run by push-button monkeys.