[Owasp-portland] Evaluating Penetration Testers

Timothy D. Morgan tmorgan-owasp at vsecurity.com
Wed May 5 02:58:42 EDT 2010

Thanks James for the insight.

I know the moment has sort of passed on this discussion, but I'll
throw a few of my own opinions out there.

I do penetration testing and vulnerability assessments regularly.
It's probably 80+ percent of what I spend my work time on.  I do both
application penetration tests and more traditional network penetration
tests and vulnerability assessments.

> He's spot on about the biggest misconception out there ("Confusion"). 
> A vulnerability scan, assessment, etc. is not the same as a pen test, yet 
> many buyers and sellers refer to a pen test when they want vulnerability 
> testing (and this is a big peeve of mine). A vulnerability test/scan/etc.
> means running tools (like Nessus or a long list of commercial tools) to
> find vulnerabilities - without fully exploiting them. This means no damage
> to systems or data, typically no denial of service attacks, just inferences
> from the testing about the problems that exist and their severity.
> Many more firms offer this type of service. Many of them just give you 
> the scanning tool output with minimal added value, leaving interpretation, 
> remediation details, risk quantification, etc. to the buyer; my personal 
> advice is to look at sample deliverables and weed out those particular
> providers, unless that's really what you want (in which case, why not do 
> it yourself?). A true pen test involves acquiring intelligence, through
> scanning tools and the like, in order to launch a targeted attack which
> succeeds in either acquiring or destroying data or availability of it.
> That targeted attack may take some time and testing (on your own - not 
> against the intended target) to get right.

I think this is a very good way to look at it.  There is indeed a lot
of confusion out there (mostly created by people trying to make what
they're selling sound better than it is) about what a vulnerability
assessment is and what a penetration test entails.  This is also a
frustration of mine, and I'm glad to see you draw the line between the
two in much the same way I would.

> Firms that are capable of conducting a REAL quality pen test are few and far 
> between.  It takes a certain skill set, a large R&D budget and lab to stay 
> on top of the skills and tools required to do real pen testing. The really 
> reputable firms dedicate staff to this with really good backgrounds; less
> reputable ones might hire someone whose hat is a much darker shade of gray 
> who you probably don't want trying to break in to your network. There's 
> another category of tinkerers who think they've got what it takes from
> playing around with metasploit; you don't want them either because they're 
> far less likely to succeed - and not because your defenses are top notch.

There are a lot of relatively inexperienced consultants out there
providing vulnerability assessments.  To add to this confusion, many
large consulting firms subcontract to smaller companies to provide the
technical muscle.  While this may often work out fine (I work for such
a smaller company), it's hard to know if from project to project
where you'll receive consultants from.  (I guess that's true even if
there is no subcontracting going on.)  I think for all of these
reasons, it's useful to get as much information as you can about who
will be delivering the work.

> Next, few firms that want a "pen test" are really in a position to 
> derive value from an actual pen test; they want a report card that doesn't
> involve a potential denial of service or actual theft of credentials
> or data.

While the risk of untrustworthy pentesters always exists, I don't
think it's all that significant.  I think in the 90's, companies were
much more receptive to the idea of taking some hacker kid and turning
him around to the "good side", but the honeymoon is long over on that
idea.  (Just take a look at Mitnick's saga.)  So if a pentester has a
shady background, they're either hiding it really well (as could any
employee), or they're having a hard time finding work.  I would guess
a "smart shady" character wouldn't want to risk blowing their cover by
actually doing evil things to systems they were already testing, but
then I guess there's a level of irrationality there too...

> If the point of the test is not to simulate an actual intrusion, and potentially
> succeed, one should really question what they want to get out of testing.
> If the point (as it often appears to be) is to "scare" executives into
> releasing $$ because you have been otherwise unsuccessful in getting $$ and
> attention on security issues you already know about, I strongly encourage you 
> to find another way to get that message across (plug: I'm here to help :).

When I first started penetration testing, I was somewhat surprised by
how much I was encouraged to fully exploit certain flaws, rather than
just note them and provide recommendations to fix them.  However, over
time I began to realize that fully exploiting problems does have a
number of benefits.  

Obviously for the consultant, there's a "wow factor" that goes along
with breaking into something.  It helps prove our competence, but yes,
it can easily turn into a trophy hunt.  However, wow factor can be
useful for our clients too.  If critical data was accessed, this may
help get funds allocated more toward security within the organization.

Another, perhaps more important, benefit is to fully understand the
issue and the difficulty of exploiting it.  Perhaps on the surface a
particular bug looks easy to exploit, but 5 hours later, you realize
it's tough to do in practice and you can perhaps reduce it's severity.

The same goes the other way... I've been noticing an unpublished bug
in a popular piece of commercial software for a few years now that I
considered potentially serious, but probably not easily exploitable.
I finally got around to fully exploiting it last week and realized
that it's really bad and not that hard to exploit.  Most customers
wouldn't have expected me to go the extra mile and exploit this flaw,
given our typical engagement parameters, but now do I feel stupid for
not trying a little harder earlier.  Well, at least I can go back
through my old reports and notify all of my customers about how to
better prioritize it.  

In many cases though, especially in traditional network-level tests,
that doing full penetration testing often doesn't add a great deal of
value.  On the application side, the automated tools work far less
well and you're regularly coming across flaws which are novel and hard
to assess, so exploiting them there has a lot more value.

> One of the big downsides to a true blind (no knowledge) pen test is
> alienation of the IT staff. The most common desired end state is improved 
> infrastructure and application security; think of the test from the perspective of the IT staff that owns it: if someone is brought in without your knowledge
> and trashes your stuff, how likely are you to be open minded to working 
> with them to help you fix the problems they found? It is set up to create
> an adversarial relationship, much more so than a vulnerability test.
> In addition to the IT staff point of view, it's a bad metric of overall
> security; it's a trophy hunt. A reputable, skilled firm (back to who is 
> capable of a real pen test) will likely succeed, and now you probably have a 
> pissed off IT staff, and a trophy to show the C-level guys. You have found 
> and exploited one or more vulnerability, but you don't have a general 
> report card on the state of your IT assets. You may succeed in getting 
> money and attention on the problem(s) that got exploited, but you don't have
> an overall remediation strategy based on risk and the big picture. 

I agree you definitely do not want an adversarial relationship with IT
staff.  In addition, I rarely see any benefit in doing blind
penetration tests.  Typically we work pretty closely with IT staff to
arrange testing, but occasionally we get customers who feel that it
isn't a "good test" of their environment if we're given any
information about it.  Usually those customers are just wasting my
time and their money.  Chances are there are much better things to be
spending scarce security funds on than playing war games.  I can
provide a lot more benefit if you show me how things are roughly put
together first.  Yes, that gives me an advantage over an attacker, but
only by a day or two of recon.

If you got this far, thanks for reading. =)

More information about the Owasp-portland mailing list