[Owasp-board] Community engagement and certification

Jim Manico jim.manico at owasp.org
Fri Apr 4 03:59:59 UTC 2014

(comments on certification from the field)

The problem with most of the certifications is some (previously) clueless person goes to bootcamp to do an exam-cram for a week (or 2, at most) and then go take the exam and pass it.

Having taught a master's level CS course in Computer Security for 6 years (probably about 8 or 9 semesters as sometime I taught it 2x per year), I can tell you that multiple choice questions do little to test comprehension. (The SAT and ACT folks may argue a bit, but even then require problem solving, not just memorization.)

While multiple choice questions are easy to grade (can be completely automated using something like Scanatron), IMO, they are terrible at teaching understanding. Thus we see tests such as the ones ISC^2  administers (and somewhat less of a degree with SANS) questions that are obscured with
double or triple negatives and other trick type questions. If they didn't do that, way too many people would pass.

To really test comprehension of subject matter, you need to pose a problem and then have someone solve it. (I've been told that SANS does this with some of their tests, although GWEB wasn't one of those; if so, good for them.) Generally, you want them to write out their answer, which means at least short answer, or perhaps essay questions. Unfortunately, those are somewhat subjective to grade.

An alternative might be provide multiple test answers and a working demo similar to WebGoat that they can test against and ask them to find a particular SQLi or XSS.

The answers of course couldn't be so specific as to show allow them to test each possible answer, but rather something generic like (for an SQLi question):
a) It is possible to insert a new admin user into the Users table.
b) It is possible to dump the Users table along with the hashed password.
c) It is possible to execute a command on the DB server.
d) It is possible to bypass the login form using SQLi.

That at least has an objective answer to it, unlike short answer / essay.  The problem in my experience is that those types of really good questions take about an order of magnitude to come up with.

You could also show code snippets and ask questions about "what is wrong with this code?" (as in what type of an attack does it allow) and "which of these is the proper fix for xyz attack on this code snippet?", etc.

It won't be easy, but you just MIGHT end up with an exam that those who are already experts have confidence in that someone who can pass it has at least some level of understanding about security. (I personally cannot attest to that for CISSP.)

Also, I don't think we should be as broad as ISC^2's CISSP and there 12 security domains of knowledge. IRL, no one needs to be experts on all 12 of those. (When was the last time you had to make a recommendation for a standing pipe fire suppression unit or provide legal advice for Grahm-Leach-Bliley? Thought so.)

Instead, we have centered OWASP around builders, breakers, and defenders. Those areas are highly specialized. I know experts whom I respect that work in one area that couldn't pass a test in the other two areas and that's okay. The more specialized in-depth knowledge you get the more you have to ignore outside your fields. The idea of a plug-compatible member of technical staff is a management myth in information security engineering just as much as it is in any other engineering discipline.


On 4/3/14, 8:21 PM, Jim Manico wrote:
> Board,
> If the foundation wants to do a certification program, here are my 
> rough notes on the topic:
> *Managing Questions**
> *1) We need a question bank working group of experts, possible paid, 
> to create and maintain certification questions.
> 2) These questions need to be kept in secret
> 3) None of the question bank workers can apply for the certification. 
> Being a question writer should be honored, but they cannot get the 
> cert itself.
> *Which Certs**
> *1) I say we start small with one assessment centric cert since this 
> is primarily a pentester organization
> *How to deliver the test**
> *1) We can use a formal testing center like 
> http://www.sylvanlearning.com/ who takes a cut from the cert to 
> deliver it.
> *Reality**
> *2) We probably need a FTE to manage this process.
> *Bonus Points**
> *1) Bonus points: it would be nice to find a PhD in Education or 
> similar who can help us do this with academic integrity.
> 2) Bonus points: get the cert ANSI certified for even broader 
> acceptance in some govt and large company communities.
> Just a start,
> Aloha,
> Jim
>> Board,
>> Where are we at with our investigation into certifications and 
>> community engagement? I realize we're just pulling together 
>> information on the topic at the moment but I was curious on the status.
>> Jim - did you have the lead here? Can you provide an update?
>> https://www.owasp.org/index.php/March_3,_2014#New_Business
>> --
>> Michael Coates
>> @_mwc
>> _______________________________________________
>> Owasp-board mailing list
>> Owasp-board at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-board

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20140403/55ff7aee/attachment.html>

More information about the Owasp-board mailing list