[Owasp-cert] Philosophies on Exam Creation

J. Oquendo sil at infiltrated.net
Sun Jul 20 11:29:23 EDT 2008

On Sat, 19 Jul 2008, Gary Palmer wrote:

> 1- Question that ask "which of the following do NOT..." are good because
> they help identify if a user can understand a grouping of items.  For
> example, if you were proving to me that you could be an auto mechanic and
> had to pass a test, a reasonable question might be "which of the following
> parts are not found on a commercial car?  1) actuator, 2) fuel injector, 3)
> throttle, 4) 30mm Cannons"  But the choice, being ridiculous, make it a
> simple solution and a useless question.  But if you trade #4 with, "sump
> pump" or "bilge pump" you might catch a few who do not know the parts.  Then
> the argument is "cars are so complicated, how should someone be expected to
> know all the parts", but we are asking them to prove knowledge, not ability
> to Google.  When all the choices sound good it is a tough test to find the
> incorrect one.  That is why many intelligence tests ask you to pick the one
> that does not belong with the others.
> On the other hand, this type of question is harder to make work and easier
> to mess up in it's creation.  Having a bad question is no helpful either.
> This would indicate that a "plea" system (like CISSP or SANS) could help, if
> you feel a question is unfair or confusing, then you have a venue to argue
> your case.

I concure to an extent with these kinds of questions. Taking it slightly

A _____ can protect against a CSRF attack:
a) WAF
b) ALG
c) IDS
d) IPS

While you may tweak an IPS to protect against those, the exam is being based
on Web Applications therefore the absolute answer would be WAF. And no, I
would not describe WAF, ALG, IPS etc. Someone taking this exam should be
familiar enough with the terminology pertaining to an exam to know that ATM
is NOT "At This Moment" but Async. Transfer Mode if dealing with networking.

The answers should all be somehow relative and true with solely one being
the absolute when it comes to OWASP.

> 3- Are you saying that all questions and the answers to each question should
> be randomized?  Or maybe groups of related questions should be together?  I
> like the idea of groupings, like all questions about network intrusion are
> groups together randomly sequenced and answers to each questions randomly
> ordered.  We can even randomize the order the groups appear, but I do think
> groupings is useful to help people allocate time for the test.  That begs
> another question, will test takers be allowed to go back to prior questions?
> Will the test be administered on paper or electronically?

I dislike the grouping method for the following reason. Memorization. Its easy
to throw together a range of index cards and memorize them right before the
exam. When you think of security I've found its often a better method to see
some form of chaos, randomness, it keeps one on their toes and weeds out some
of the pros from the joes.

Random answers all pertaining to OWASP in general while maintaining a focus
on the industry from my point of view will make those passing the test more
apt to not only read, but understand it. Even if say a book comes out as a
study guide for the OWASP exam, I believe a random question not discussed
should be thrown in from time to time to test whether or not someone got
the concepts down to a science.

I interview people in my company from time to time, and one of my favorite
questions for say Linux/Security staff is: "You're under attack from a host
at, how do you stop it without using your firewall?" I do this to
see if they're going to tell me about /etc/hosts.deny route, etc., and to
date I've had one person give me a correct answer. I can think of 4 different
ways to do this without a Linux firewall. For those who haven't seen Pete
Herzog's "Jack Of All Trades" security certification questions, its a very
interesting concept: E.g. Name 10 different ways to turn off the light without
touching the switch...

> Now with all that said, there are a couple other things to ferment:
> There should be a "normal distribution" between easy and difficult
> questions.  This bell curve slides right or left until we are getting a
> desired percentage passing the test.  If everyone passes, the test is too
> easy and the credential loses credibility.  If the test is too hard, fewer
> people take it and the test becomes elitist and distained.

Depends on your view. I know of CCNP's who outsmart CCIE's while we all know
the CCIE is the epitome of networking (to a degree). I believe the test can
be constructed with enough tough to medium questions to make it viable to
pass for someone with say 2 years hands on experience. This would be the
equivalent I guess to someone going for an associates perhaps. The structure
would/should be enough to make the exam something one would need to have
hands on experience along with the book reading to pass.

It would be nicer even if say, two or three questions focused on a specific
were say a packet dump showing some CSRF/XSS/*Other* in detail and one had
to look at the results and answer on the cause of it... Or, even a question
or two on a specific *random/vendor neutral* tool's output:

The following X was captured using which of the following:
a) Webscarab
b) W3AF
c) so on
d) so on

> I guess I am looking at a top down approach.  Again, I am not trying to
> counter what James states, just offer another perspective and ask for
> consideration of several other very relevant topics.  Thank you for your
> time.

I guess I'll unload the notion of an "expert/pentest/OWASP" bonus of say
realizing an attack after an exam, similar to say the CCE exam where in
order to obtain X_Bonus status, a virtual machine may have randomized
vulnerabilities, candidate need now 1) put their knowledge to the test
and either protect or attack constructed machine using predefined templates
of sorts along with an explanation of why they took said approach (think
SANS' Challenge This)

Have a good weekend.

J. Oquendo
SGFA #579 (FW+VPN v4.1) SGFE #574 (FW+VPN v4.1)

"Experience hath shewn, that even under the best
forms (of government) those entrusted with power
have, in time, and by slow operations, perverted
it into tyranny." Thomas Jefferson

wget -qO - www.infiltrated.net/sig|perl


More information about the Owasp-cert mailing list