Penetration Testing Methodology: Fact or Fiction?

I had a blog reader over the weekend shoot me an email asking me about my opinion on what the best penetration testing methodolgy is. I've performed a couple of hundred penetration tests, and I can say that over the last couple of years I've been exposed to (maybe I should say subjected to?) about a dozen different methodologies at different points in time. (BTW: If you have something you'd like me to write about, I'm open to suggestions!) I've seen both in-house and "industry standard" methodologies. If you aren't familiar with penetration testing methodologies, the idea behind them is that the penetration tester should follow a pre-scripted format for orchestrating the test as dictated by the methodology. Here are some of the 3 popular ones that come to mind:

The OSSTMM
NIST 4-Stage Pen-Testing Guideline
FoundStone's Pen-Testing Methodology

Now, what the reader was hoping for was a succinct, clear cut, answer. "Go here, download this, print it out, done, etc.". As you might have already guessed, I'm not going to give that level of satisfaction. And, I'm asked this question almost weekly, so now I'll be able to point people to this blog posting for the long winded answer. Ok, on to the answer.

In a nutshell, I firmly believe that any penetration testing methodology, no matter how well thought out, has limited usefulness. Why? Well the goal behind penetration testing is to try to find as many serious vulnerabilities as possible. In order to do this, you must develop the "mindset" of your attacker. You should look at your assessed system or application in all of the possible ways you think it could be misued, abused and exploited. You should then take a break, drink some well-deserved coffee, and then think of entirely new "misuse cases" for the system under review. Using a cut and dry methodology runs counter to the basic and essential premise of penetration testing; that a penetration test is an exercise in system abuse and cannot be readily scripted.

I realize that if you have never performed a penetration test, and don't have the faintest idea where to begin, you might get some value of out a methodology. However, I would venture to say that your time would be better spent hacking away on some dedicated lab equipment, writing your first Metasploit module, or writing a proposal for your boss to send you to a decent penetration testing course.

I also realize that the other (and in my eyes, more legitmate) reason to use a penetration testing methodology is the CYA factor. Most managers don't like the idea of employees willy-nilly hacking into things. The idea that an "industry standard methodology is being applied in accordance to best practices" sounds a lot better to the person in the corner office. Then, the methodology becomes more of a documentation tool, which I do see real value in.

Before I get too positive on documentation, remember what we have seen with vulnerability scanners. Generating lots of documentation can equally as dangerous. Everyone knows what has happened in the last 5 years with "vulnerablity assessment reports" generated out of nessus/internet scanner/etc. We all know the process:

1. Your boxes are rooted by 16 yr old dude in norway, he uses them to serve phising bait to 100 million paypal users.
2. High priced consultants run nessus for you, charge your company $50,000, take you out for drinks one night, generate a monster 500 page report.
3. So many false positivies and false negatives in the report no real vulns are ever acted on
4. Next month the same kid has your Oracle production E10000 serving PsyBNC to half of the norweigan underground.

Documentation can be paralyzing. It can be useless and point people down the wrong path, with end effect of people losing faith in the entire assessment process.

To sum up: Don't use a methodology unless you need to for documentation purposes, if you do, make sure the reports you generate deliver actionable intelligence on the security posture of the asssessed system.

Thoughts?

Jack