[Owasp-portland] Evaluating Penetration Testers

James C. Bohem james at holycow.portland.or.us
Mon Apr 19 15:09:30 EDT 2010

On Apr 12, 12:05pm, Timothy D. Morgan wrote:
} Subject: [Owasp-portland] Evaluating Penetration Testers
} I just saw this relatively well thought-out posting on other mailing
} lists and figured I would break the silence here and share:
}   http://seclists.org/pen-test/2010/Apr/22
} Anyone have any strong opinions they'd like to add?

There haven't been any replies... anyone out there? I'll wade in, with the 
caveats that I am definitely very opinionated about this, and I subscribe
to this list and am responding from my home email, as these opinions may
or may not reflect my employers' point of view on the topic. I also 
represent a potential seller of services, but I'm not making a sales pitch.

First, if you haven't read the thread Tim refers to above, you should; at
least the original post by Daniel Kennedy.

He's spot on about the biggest misconception out there ("Confusion"). 
A vulnerability scan, assessment, etc. is not the same as a pen test, yet 
many buyers and sellers refer to a pen test when they want vulnerability 
testing (and this is a big peeve of mine). A vulnerability test/scan/etc.
means running tools (like Nessus or a long list of commercial tools) to
find vulnerabilities - without fully exploiting them. This means no damage
to systems or data, typically no denial of service attacks, just inferences
from the testing about the problems that exist and their severity.
Many more firms offer this type of service. Many of them just give you 
the scanning tool output with minimal added value, leaving interpretation, 
remediation details, risk quantification, etc. to the buyer; my personal 
advice is to look at sample deliverables and weed out those particular
providers, unless that's really what you want (in which case, why not do 
it yourself?). A true pen test involves acquiring intelligence, through
scanning tools and the like, in order to launch a targeted attack which
succeeds in either acquiring or destroying data or availability of it.
That targeted attack may take some time and testing (on your own - not 
against the intended target) to get right.

Firms that are capable of conducting a REAL quality pen test are few and far 
between.  It takes a certain skill set, a large R&D budget and lab to stay 
on top of the skills and tools required to do real pen testing. The really 
reputable firms dedicate staff to this with really good backgrounds; less
reputable ones might hire someone whose hat is a much darker shade of gray 
who you probably don't want trying to break in to your network. There's 
another category of tinkerers who think they've got what it takes from
playing around with metasploit; you don't want them either because they're 
far less likely to succeed - and not because your defenses are top notch.

Next, few firms that want a "pen test" are really in a position to 
derive value from an actual pen test; they want a report card that doesn't
involve a potential denial of service or actual theft of credentials or data.
Firms that really are ready for a pen test have a thorough incident response
plan, individuals trained to follow that plan, some forensics abilities, 
and well configured and monitored intrusion detection capabilities. 
If the point of the test is not to simulate an actual intrusion, and potentially
succeed, one should really question what they want to get out of testing.
If the point (as it often appears to be) is to "scare" executives into
releasing $$ because you have been otherwise unsuccessful in getting $$ and
attention on security issues you already know about, I strongly encourage you 
to find another way to get that message across (plug: I'm here to help :).

One of the big downsides to a true blind (no knowledge) pen test is
alienation of the IT staff. The most common desired end state is improved 
infrastructure and application security; think of the test from the perspective of the IT staff that owns it: if someone is brought in without your knowledge
and trashes your stuff, how likely are you to be open minded to working 
with them to help you fix the problems they found? It is set up to create
an adversarial relationship, much more so than a vulnerability test.
In addition to the IT staff point of view, it's a bad metric of overall
security; it's a trophy hunt. A reputable, skilled firm (back to who is 
capable of a real pen test) will likely succeed, and now you probably have a 
pissed off IT staff, and a trophy to show the C-level guys. You have found 
and exploited one or more vulnerability, but you don't have a general 
report card on the state of your IT assets. You may succeed in getting 
money and attention on the problem(s) that got exploited, but you don't have
an overall remediation strategy based on risk and the big picture. 

I could go on and on... if folks out there want evaluation criteria for
buying either a pen test or vulnerability test, I can elaborate on that.

I've done many vulnerability tests and a handful of true penetration tests,
and I focus primarily on application testing in this space. I've been
involved in security for 20+ years and have a development background
in UNIX internals (kernel, system libraries, network stack, etc.) and 
large scale distributed applications.

James Bohem

More information about the Owasp-portland mailing list