[Owasp-board] Community engagement and certification

Tobias tobias.gondrom at owasp.org
Tue Apr 8 09:19:07 UTC 2014


I definitely agree on the aligning of questions and material.

And I support Jim's points.
And like the idea about "builders, breakers, and defenders"...

A few small comments when writing exam questions:
(disclaimer: I am in the exam writer pool of another org)

- some standards now do ask not to do double negatives in questions or
trick questions anymore
- there is a tendency to use "understanding questions" (over simple
knowledge ones) and evaluate them via multiple choice.
- writing good exam questions is not easy and costs a lot of time....

Best, Tobias


On 07/04/14 21:52, Eoin wrote:
> Given we are publishing training materials Id suggest aligning
> questions to training and cheat sheets...
>
>
>
> On 4 April 2014 04:59, Jim Manico <jim.manico at owasp.org
> <mailto:jim.manico at owasp.org>> wrote:
>
>     (comments on certification from the field)
>
>     The problem with most of the certifications is some (previously) clueless person goes to bootcamp to do an exam-cram for a week (or 2, at most) and then go take the exam and pass it.
>
>     Having taught a master's level CS course in Computer Security for 6 years (probably about 8 or 9 semesters as sometime I taught it 2x per year), I can tell you that multiple choice questions do little to test comprehension. (The SAT and ACT folks may argue a bit, but even then require problem solving, not just memorization.)
>
>     While multiple choice questions are easy to grade (can be completely automated using something like Scanatron), IMO, they are terrible at teaching understanding. Thus we see tests such as the ones ISC^^2  administers (and somewhat less of a degree with SANS) questions that are obscured with
>     double or triple negatives and other trick type questions. If they didn't do that, way too many people would pass.
>
>     To really test comprehension of subject matter, you need to pose a problem and then have someone solve it. (I've been told that SANS does this with some of their tests, although GWEB wasn't one of those; if so, good for them.) Generally, you want them to write out their answer, which means at least short answer, or perhaps essay questions. Unfortunately, those are somewhat subjective to grade.
>
>     An alternative might be provide multiple test answers and a working demo similar to WebGoat that they can test against and ask them to find a particular SQLi or XSS.
>
>     The answers of course couldn't be so specific as to show allow them to test each possible answer, but rather something generic like (for an SQLi question):
>     a) It is possible to insert a new admin user into the Users table.
>     b) It is possible to dump the Users table along with the hashed password.
>     c) It is possible to execute a command on the DB server.
>     d) It is possible to bypass the login form using SQLi.
>
>     That at least has an objective answer to it, unlike short answer / essay.  The problem in my experience is that those types of really good questions take about an order of magnitude to come up with.
>
>     You could also show code snippets and ask questions about "what is wrong with this code?" (as in what type of an attack does it allow) and "which of these is the proper fix for xyz attack on this code snippet?", etc.
>
>     It won't be easy, but you just MIGHT end up with an exam that those who are already experts have confidence in that someone who can pass it has at least some level of understanding about security. (I personally cannot attest to that for CISSP.)
>
>     Also, I don't think we should be as broad as ISC^^2 's CISSP and there 12 security domains of knowledge. IRL, no one needs to be experts on all 12 of those. (When was the last time you had to make a recommendation for a standing pipe fire suppression unit or provide legal advice for Grahm-Leach-Bliley? Thought so.)
>
>     Instead, we have centered OWASP around builders, breakers, and defenders. Those areas are highly specialized. I know experts whom I respect that work in one area that couldn't pass a test in the other two areas and that's okay. The more specialized in-depth knowledge you get the more you have to ignore outside your fields. The idea of a plug-compatible member of technical staff is a management myth in information security engineering just as much as it is in any other engineering discipline.
>
>     ***
>
>     On 4/3/14, 8:21 PM, Jim Manico wrote:
>>     Board,
>>
>>     If the foundation wants to do a certification program, here are
>>     my rough notes on the topic:
>>
>>     *Managing Questions**
>>     *1) We need a question bank working group of experts, possible
>>     paid, to create and maintain certification questions.
>>     2) These questions need to be kept in secret
>>     3) None of the question bank workers can apply for the
>>     certification. Being a question writer should be honored, but
>>     they cannot get the cert itself.
>>
>>     *Which Certs**
>>     *1) I say we start small with one assessment centric cert since
>>     this is primarily a pentester organization
>>
>>     *How to deliver the test**
>>     *1) We can use a formal testing center like
>>     http://www.sylvanlearning.com/ who takes a cut from the cert to
>>     deliver it.
>>
>>     *Reality**
>>     *2) We probably need a FTE to manage this process.
>>
>>     *Bonus Points**
>>     *1) Bonus points: it would be nice to find a PhD in Education or
>>     similar who can help us do this with academic integrity.
>>     2) Bonus points: get the cert ANSI certified for even broader
>>     acceptance in some govt and large company communities.
>>
>>     Just a start,
>>     Aloha,
>>     Jim
>>
>>>     Board,
>>>
>>>     Where are we at with our investigation into certifications and
>>>     community engagement? I realize we're just pulling together
>>>     information on the topic at the moment but I was curious on the
>>>     status.
>>>
>>>     Jim - did you have the lead here? Can you provide an update?
>>>
>>>     https://www.owasp.org/index.php/March_3,_2014#New_Business
>>>
>>>
>>>
>>>     --
>>>     Michael Coates
>>>     @_mwc
>>>
>>>
>>>
>>>     _______________________________________________
>>>     Owasp-board mailing list
>>>     Owasp-board at lists.owasp.org <mailto:Owasp-board at lists.owasp.org>
>>>     https://lists.owasp.org/mailman/listinfo/owasp-board
>>
>
>
>     _______________________________________________
>     Owasp-board mailing list
>     Owasp-board at lists.owasp.org <mailto:Owasp-board at lists.owasp.org>
>     https://lists.owasp.org/mailman/listinfo/owasp-board
>
>
>
>
> -- 
> Eoin Keary
> OWASP Member
> https://twitter.com/EoinKeary
>
>
>
> _______________________________________________
> Owasp-board mailing list
> Owasp-board at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-board

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20140408/967431f1/attachment.html>


More information about the Owasp-board mailing list