[Esapi-user] Gary McGraw informIT article on BSIMM

Kevin W. Wall kevin.w.wall at gmail.com
Tue Feb 2 21:38:26 EST 2010

Hi folks,

Gary McGraw, from Citigal, wrote an editorial sort of article advocating
BSIMM (Building Security In Maturity Model, http://www.bsi-mm.com/). It is
relevant to these lists because it contrasts and advocates BSIMM's descriptive
model to ESAPI's prescriptive approach.

I originally had intended posting this only to the Secure Coding List (SC-L),
but I was convinced by a certain someone that it perhaps would be pertinent
to these 2 ESAPI lists as well. For those of you who subscribe to both lists,
I apologize in advance for the cross-posting. To avoid duplicate replies, please
follow-up to the ESAPI-Users list only.

In case it's not obvious, SSG is "software security group", a group dedicated
specifically to software security within a company.


On Thu, 28 Jan 2010 10:34:30 -0500, Gary McGraw wrote:

> > Among other things, David [Rice] and I discussed the difference between
> > descriptive models like BSIMM and prescriptive models which purport to
> > tell you what you should do.  I just wrote an article about that for
> > informIT.  The title is
> >
> > "Cargo Cult Computer Security: Why we need more description and less
> > prescription."
> > http://www.informit.com/articles/article.aspx?p=1562220

First, let me say that I have been the team lead of a small Software
Security Group (specifically, an Application Security team) at a
large telecom company for the past 11 years, so I am writing this from
an SSG practitioner's perspective.

Second, let me say that I appreciate descriptive holistic approaches to
security such as BSIMM and OWASP's OpenSAMM. I think they are much
needed, though seldom heeded.

Which brings me to my third point. In my 11 years of experience working
on this SSG, it is very rare that application development teams are
looking for a _descriptive_ approach. Almost always, they are
looking for a _prescriptive_ one. They want specific solutions
to specific problems, not some general formula to an approach that will
make them more secure. To those application development teams, something
like OWASP's ESAPI is much more valuable than something like BSIMM or
OpenSAMM. In fact, I confirm that your BSIMM research would indicate that
many companies' SSGs have developed their own proprietary security APIs
for use by their application development teams. Therefore, to that end,
I would not say we need less _prescriptive_ and more _descriptive_
approaches. Both are useful and ideally should go together like hand and
glove. (To that end, I also ask that you overlook some of my somewhat
overzealous ESAPI developer colleagues who in the past made claims that
ESAPI was the greatest thing since sliced beer. While I am an ardent
ESAPI supporter and contributor, I proclaim it will *NOT* solve our pandemic
security issues alone, nor for the record will it solve world hunger. ;-)

I suspect that this apparent dichotomy in our perception of the
usefulness of the prescriptive vs. descriptive approaches is explained
in part by the different audiences with whom we associate. Hang out with
VPs, CSOs, and executive directors and they likely are looking for advice on
an SSDLC or broad direction to cover their specifically identified
security gaps. However, in the trenches--where my team works--they want
specifics. They ask us "How can you help us to eliminate our specific
XSS or CSRF issues?", "Can you provide us with a secure SSO solution
that is compliant with both corporate information security policies and
regulatory compliance?", etc. If our SSG were to hand them something like
BSIMM, they would come away telling their management that we didn't help
them at all.

This brings me to my fourth, and likely most controversial point. Despite
the interesting historical story about Feynman, I question whether BSIMM
is really "scientific" as the BSIMM community claims. I would contend
that we are only fooling ourselves if we claim otherwise. And while
BSIMM is a refreshing approach opposed to the traditional FUD modus
operandi taken by most security vendors hyping their security products,
I would argue that BSIMM is no more scientific than the those
who gather common quality metrics of counting defects/KLOC. Certainly
there is some correlation there, but cause and effect relationships
are far from obvious and seem to have little predictive accuracy.

Sure, BSIMM _looks_ scientific on the outside, but simply collecting
specific quantifiable data alone does not make something a scientific
endeavor.  Yes, it is a start, but we've been collecting quantifiable
data for decades on things like software defects and I would contend
BSIMM is no more scientific than those efforts. Is BSIMM moving in
the right direction? I think so. But BSIMM is no more scientific
than most of the other areas of computer "science".

To study something scientifically goes _beyond_ simply gathering
observable and measurable evidence. Not only does data needs to be
collected, but it also needs to be tested against a hypotheses that offers
a tentative *explanation* of the observed phenomena;
i.e., the hypotheses should offer some predictive value. Furthermore,
the steps of the experiment must be _repeatable_, not just by
those currently involved in the attempted scientific endeavor, but by
*anyone* who would care to repeat the experiment. If the
steps are not repeatable, then any predictive value of the study is lost.

While I am certainly not privy to the exact method used to arrive at the
BSIMM data (I have read through the "BSIMM Begin" survey, but have not
been involved in a full BSIMM assessment), I would contend that the
process is not repeatable to the necessary degree required by science.
In fact, I would claim in most organizations, you could take any group
of BSIMM interviewers and have them question different people in the
organization using the exact same questions and arrive at different results.
In fact, I am willing to bet that the different members of my Application
Security team who have all worked together for about 8 years would
answer a significant number of the BSIMM Begin survey questions quite
differently. (My explanation for this phenomena is the general observation
that if you ask a group of N engineers for their opinion on something,
they will almost certainly arrive at N+1 different opinions. ;-)

I commend the BSIMM sponsors and leadership of releasing BSIMM under
a Creative Commons license, but at the same time, I challenge them
to put forth additional information explaining their data collection
process and in particular, describing how it
avoids unintentional bias. (E.g., Are assessment participants choose at
random? By whom?  How do you know you have a representative sample of
a company? Etc.)

I also challenge BSIMM to show their data collection is repeatable by
others following their process. The published BSIMM Begin survey is a
good start, but information regarding the full assessment seems to be

I challenge BSIMM to put forth their hypotheses, plainly stated.
In your InformIT article, you wrote:

    "Another distinct advantage that descriptive models have over
    prescriptive models is the ability to compare current observations
    with past observations."

In my opinion, comparison of observations from two companies is not
worth the paper that is printed on UNLESS we can extrapolate from
this data and make accurate predictions based on past findings. Therefore,
I also would challenge BSIMM to publicly make some specific predictions
using their model and collected data so that their hypotheses can be
tested independently by others.

Finally, while I would like, as you did, to blame our Computer Security /
software security's "Cargo Cult" mentality on "its relative youth as
a field", I believe there is something deeper going on here. For one,
computer science / IT / whatever you want to call this much broader
field has the same issue. And while computer "science" is young as
measured against most other scientific disciplines, it is by no means an
immature field. (As a discipline, it is much older than I, and trust me,
I am no spring chicken. :)

After observing this field for 30+ years (ouch!), I have concluded that we
have not matured into a science because as a discipline we *do NOT
really want to!*  We can't even decide if we want this study of computers
/ information processing / etc. to be a "science", an "engineering
discipline", or a "craft". (And some even would like it to be an "art".)
Most of us--myself included--are too lazy to do the disciplined work
that true science requires, and that includes having enough guts to
challenge the academic culture and *demand* funding to do well-designed
scientific experiment in our discipline at our leading universities. IMO,
if we fail to do this, CS is doomed to always remain a science wannabe.

Some would say that because our broader field of study--whether you
call it Computer Science, Computer Engineering, Information Science,
whatever--is part science, part engineering, part craft, and part
something unique that humanity has never before encountered, attempts
to treat it as a science will not succeed. However, surely this does
not mean that we should not attempt to add some scientific rigor to
it as a discipline. To that end efforts such as BSIMM should be welcomed
by all.  But is also important for those who prefer _descriptive_ approaches
like BSIMM, to acknowledge the importance of _prescriptive_ approaches
such as ESAPI, WAFs, anti-malware software, etc. We truly need *both*
approaches to be successful.

Kevin W. Wall
"The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents."        -- Nathaniel Borenstein, co-creator of MIME

More information about the Esapi-user mailing list