[Owasp-webcert] Assurance Levels
jeff.williams at aspectsecurity.com
Mon Jun 11 11:28:05 EDT 2007
Totally agree that the criteria should have assurance levels!
But I'm not comfortable with the idea that there is a strict hierarchy
of techniques. Despite much marketing to the contrary, code review is
often the MOST cost-effective technique for verifying a particular
security area. To be perfectly clear, when I say cost-effective I mean
that for whatever amount of money you have to spend, it's faster and
more accurate to check the code. I'd like to see a model where the most
appropriate technique (or combination of techniques) can be selected to
verify the particular thing being tested. Frequently this depends on
the particular application.
I think of the problem in terms of breadth and depth. Breadth is the
coverage of security-critical areas in an application. Depth is the
level of rigor checking each of the areas. At each assurance level,
both breadth and depth increase. We had a partial attempt at coming up
with some names for a set of levels a year ago that might be interesting
Also -- in principle, I like the idea of including SDLC evidence as part
of the various levels, as it does help with assurance (somewhat
indirectly). A system to measure organizational application security
capability will end up sort of CMM like. If organizations could get
some kind of credit for an organizational capability measurement, that
would go a long way towards encouraging them to do the right things.
But creating an organizational assessment model and process that can't
be gamed (like organizations claiming CMM Level 5 when only one small
team achieves it) are pretty significant undertakings. Maybe if we
could keep it to the simplest things it would make sense to include. See
From: owasp-webcert-bounces at lists.owasp.org
[mailto:owasp-webcert-bounces at lists.owasp.org] On Behalf Of Mark Curphey
Sent: Monday, June 11, 2007 4:27 AM
To: owasp-webcert at lists.owasp.org
Subject: [Owasp-webcert] Assurance Levels
I propose to make assurance levels an integral part of the OWASP Web
Certification Criteria and want your feedback on the concept.
In many ways it's one of those things that's so damn obvious when you
see it described with clarity. Enlightenment came for me when Chris
Wysopal <http://www.veracode.com/management-team.php#Wysopal> sent me
the fantastic graphic atttached describing Veracodes
<http://www.veracode.com/> view of assurance levels. Of course it is
nothing new, the basic concept of assurance (confidence) is as follows;
Different testing techniques provide different levels of
assurance (confidence) on claims about the security of a web site.
An automated static analysis tool will provide a lower level of
assurance than an automated dynamic analysis tool which will in-turn
provide a lower level of assurance than a comprehensive manual code
review. It also follows that an automated web application penetration
test will provide a lower level of assurance than a manual penetration
test. Both types of penetration testing will provide lower levels of
assurance than code reviews. It also makes sense that if a company has
demonstrated that security is an integral part of the security DNA of
their SDLC (define, design, develop, deploy and maintain) then there is
a higher level of assurance that any test results will be consistent in
So why wouldn't everyone just go for the approach that provides the
highest level of assurance? It's very simple, cost. The appropriate
level of assurance should be based on risk.
Of course all of this things have a butterfly effect, no two tools are
the same and no two testers are the same. Imagine a control panel with
multiple dials but where people want a single output display (not
necessarily a single reading). I expect lots of people arguing that a
specific tool or firm is as good as the next level up on the assurance
level but well deal with that as well.
This also enables us to define what a web app firewall is good for, what
it isn't and place it into an assurance level bucket. More on that in a
By incorporating assurance levels into the criteria, industry sectors,
business partners or regulators can require a level of security with an
assurance level based on risk. This would be a significant step forward
from where we are today with broken schemes like PCI DSS
So what do y'all think?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Owasp-webcert