[OWASP-benchmark-project] Proposal for handling time taken for running test cases against DAST scans

Kevin W. Wall kevin.w.wall at gmail.com
Sun Dec 13 07:13:15 UTC 2015

Well, look at it from a test case writing PoV. If you know that something
is going to only be detectable by SAST or IAST, why bother to go the extra
mile to pass stuff back and forth over HTTP when DAST has a zero chance of
detecting it because it doesn't have visibility to the 'source' (as in
"source / sink")? (I.e., there are times the black box view is simply not
sufficient.) A lot of crypto test cases that I can conceive of are like
this. Building a simple test class to for the test case for SAST tools to
scan is relatively easy; creating the additional test harness code just so
DAST gets to play to (even though it can do no better than randomly guess
if an issue is present) adds additional work and seems like a waste of time
and resources to me. Sure, if there were *any* chance a black box tests
could detect if it would be worth making the extra effort, but there will
always be cases where only a white box or gray box test approach has any
chance at detecting some specific issues.

I _understand_ your point / philosophy, I just don't agree with it

Sent from my Droid; please excuse typos.
On Dec 13, 2015 12:41 AM, "Dave Wichers" <dave.wichers at owasp.org> wrote:

> All the test cases are for both. And the reason we don’t have some for
> SAST and some for DAST is so we can make direct comparisons of SAST against
> DAST tools all with a single Benchmark. To me, that’s one of the key
> features of the Benchmark in that it is the first Benchmark to allow such
> direct comparisons of SAST to DAST to IAST to any other app vuln detection
> tool because its a real runnable app.
> Some will clearly not be detectable by DAST and others will (eventually)
> not be detectable to SAST (most likely). And I don’t want to say in advance
> what type of tools can find what as vendors are getting more and more
> clever with the use of hybrid, and instrumentation, and other cool things,
> like involving other servers in the testing, etc.
> -Dave
> From: Kevin Wall <kevin.w.wall at gmail.com>
> Date: Saturday, December 12, 2015 at 10:26 PM
> To: Dave Wichers <dave.wichers at owasp.org>
> Cc: OWASP Benchmark Project <owasp-benchmark-project at lists.owasp.org>
> Subject: Re: [OWASP-benchmark-project] Proposal for handling time taken
> for running test cases against DAST scans
> Dave,
> It would seem like one useful extension that could be done would be to add
> something to the testcase###.xml files that would distinguish whether the
> particular test cases were intended for SAST, DAST, or both. That would
> allow us to drastically increase the # of SAST tests while keeping the
> longer running DAST tests at a minimum. Alternatively, they could the
> separated into distinct sub-directories. That would seem like a useful
> ability though because some 'sources' (as in source-to-sink) don't really
> apply (as in "not visible to") to (web-based) DAST tools since they only
> have a web / HTTP view.
> -kevin
> Sent from my Droid; please excuse typos.
> On Dec 12, 2015 3:02 PM, "Dave Wichers" <dave.wichers at owasp.org> wrote:
>> Kevin,
>> The test cases are generators from 3 primary building blocks:
>> 1) The source of taint (e.g., request.getParameter(), or headers, or
>> cookies, etc.)
>> 2) Various kinds of propagation constructs (including none at all).
>> 3) A sink (e.g., runtime.exec(), or an HTTP response, or an SQL query).
>> Currently, there are about:
>> 14 sources
>> Over 100 sinks
>> ~25 propagators
>> Not all of these are compatible with each other. That said, producing ALL
>> of the combinations would result in 100K or so test cases, but would
>> involve massive redundancy. That¹s why a much smaller subset is reasonable
>> to produce because each building block is included between 25 up to 100 or
>> more times already. And that¹s with the 2740 test cases.
>> I¹m OK with generating a bigger set, and in fact, it should naturally grow
>> as we add more building blocks. So I think starting with 2740 dynamic test
>> cases is a good balance between having a good / broad set, and the time it
>> takes to run a successful scan with agains the Benchmark with Dynamic
>> tools.
>> -Dave
>> P.s. We are working on a UI update for the Benchmark that will organize
>> all the test cases into different categories by vulnerability category and
>> will even break up the tests within a category as well so we never have
>> more than 80 test cases for a single directory in the URL space.  This
>> will make it easy for someone to test just one or a few specific types of
>> vulns with their DAST scanner rather than having to scan the entire
>> Benchmark each time.
>> On 12/10/15, 1:43 AM, "Kevin W. Wall" <kevin.w.wall at gmail.com> wrote:
>> >Was just reading more closely on the wiki page under the 'Test Cases'
>> >tab how it mentioned that the # of test cases have been reduced
>> >from over 20k to ~3k (specifically, 2740). The wiki page states:
>> >
>> >    Version 1.0 of the Benchmark was published on April 15, 2015
>> >    and had 20,983 test cases. On May 23, 2015, version 1.1 of
>> >    the Benchmark was released. The 1.1 release improves on the
>> >    previous version by making sure that there are both true
>> >    positives and false positives in every vulnerability area.
>> >    Version 1.2beta was released on August 15, 2015.
>> >
>> >    Version 1.2 and forward of the Benchmark is a fully
>> >    executable web application, which means it is scannable by
>> >    any kind of vulnerability detection tool. The 1.2beta has
>> >    been limited to slightly less than 3,000 test cases, to make
>> >    it easier for DAST tools to scan it (so it doesn't take so
>> >    long and they don't run out of memory, or blow up the size of
>> >    their database). The final 1.2 release is expected to be at
>> >    least 5,000 or possibly 10,000 test cases, after we determine
>> >    that the popular DAST scanners can handle that size.
>> >
>> >
>> >I was going to suggest that perhaps one way to deal with the
>> >complexity and length run times of DAST would be to all whomever
>> >is configuring the tests to have a "selector" that just allows one
>> >or more CWEs to be chosen as the test cases and then only
>> >deploy those test cases matching the selected CWEs into the
>> >mix for the DAST testing. (SAST should be fast enough to probably
>> >run against all test cases.)
>> >
>> >Note that I'm not volunteering to write the code, but one thing is that
>> >this
>> >goes back to my 1st question about contributing to the test cases.
>> >
>> >If we are not going to put all 20k of them out there, then it would be
>> >difficult to tell if some of them are redundant. And if there is a desire
>> >to put them all out there (which I believe should be the ultimate goal),
>> >then we need some better way to organize them for people to contribute,
>> >e.g., making 'cwe##' subdirectories for the test cases in
>> >    src/main/java/org/owasp/benchmark/testcode
>> >and organize them that way. (That would also make creating a 'selector'
>> >a bit easier.)
>> >
>> >Just my $.02,
>> >-kevin
>> >--
>> >Blog: http://off-the-wall-security.blogspot.com/
>> >NSA: All your crypto bit are belong to us.
>> >_______________________________________________
>> >OWASP-benchmark-project mailing list
>> >OWASP-benchmark-project at lists.owasp.org
>> >https://lists.owasp.org/mailman/listinfo/owasp-benchmark-project
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-benchmark-project/attachments/20151213/c2528f1b/attachment.html>

More information about the OWASP-benchmark-project mailing list