[Owasp-cert] Status Update

[Bil Corry]

james at architectbook.com wrote on 10/18/2008 8:00 PM: 
> 3. I think we have some good rough ideas on subject areas but believe that there 
> is something we are missing in the way of conceptualizing the design of the 
> entire program. I am attempting to figure out the "methodology" used by others 
> in this regard and most importantly getting my hands on a real-world documented 
> example.

You start out by looking at your goals, then see if your certification meets/exceeds them.  Five of the six goals, as stated in the latest draft PDF, all focus on measuring the knowledge and skills of the certification holder.  The remaining goal is to get the universities teaching secure software design and development.

Will the certification program achieve those goals?  In order to meet/exceed the goal of measuring knowledge and skill, the exam must have very specific metrics that accurately reflect the knowledge and skill of the test taker.  This is the most important piece to the certification program as no one will find it credible otherwise.  So once the exam has been created, it must be tested against people whose proficiency level is already known, that way you can tell if the exam is performing as expected, including giving the exam to people not in the webapp field, such as graphic artists or janitors etc (obviously if they score high, then the exam needs work).

There appears to be a number of areas of knowledge have been identified to test against, each with 10 to 20 questions each.  Is it possible to accurately measure someone's knowledge of one area based on 10 to 20 questions?  And is the difference between an Associate, a Professional and a Master great enough?  For example, Lifecycle has a proposed 10 questions.  If you got 0 to 6 correct, you failed that section.  If you got 7 correct, you are an Associate, 8 is a Professional and 9/10 is a Master.  So the difference between an Associate and a Professional is a single correctly answered question.  Unless the exam is written just right, that small margin of difference means some Professionals will be labeled as Associates and some Associates will be labeled as Professionals.

Some things to consider, and it may be too late for this: it may make sense to have more than one exam where each exam focuses on a few core areas; that way areas such as Lifecycle can have more than a handful of questions to distinguish between Associate and Master.  And instead of a strict certification program with three skill levels (Associate, Professional, Master), maybe approach it more like the SAT, where everyone who takes it gets a total score, then the score broken out by area -- this allows the goal of comparing skills to one another and it also allows the universities to gauge how effective their curriculum is.  Then on top of the total score, you can provide the Associate, Professional, Master certification.  And for the "Master" certification, I'd strongly suggest that they must achieve better than 90% on every area, rather than maybe 100% in Lifecycle, but 80% in Cryptography.  You want someone with a "Master" certification to really be a "Master" at webappse
c.


- Bil