[OWASP-benchmark-project] Proposal for handling time taken for running test cases against DAST scans

Dave Wichers dave.wichers at owasp.org
Sat Dec 12 20:02:27 UTC 2015


The test cases are generators from 3 primary building blocks:

1) The source of taint (e.g., request.getParameter(), or headers, or
cookies, etc.)
2) Various kinds of propagation constructs (including none at all).
3) A sink (e.g., runtime.exec(), or an HTTP response, or an SQL query).

Currently, there are about:

14 sources
Over 100 sinks
~25 propagators

Not all of these are compatible with each other. That said, producing ALL
of the combinations would result in 100K or so test cases, but would
involve massive redundancy. That¹s why a much smaller subset is reasonable
to produce because each building block is included between 25 up to 100 or
more times already. And that¹s with the 2740 test cases.

I¹m OK with generating a bigger set, and in fact, it should naturally grow
as we add more building blocks. So I think starting with 2740 dynamic test
cases is a good balance between having a good / broad set, and the time it
takes to run a successful scan with agains the Benchmark with Dynamic


P.s. We are working on a UI update for the Benchmark that will organize
all the test cases into different categories by vulnerability category and
will even break up the tests within a category as well so we never have
more than 80 test cases for a single directory in the URL space.  This
will make it easy for someone to test just one or a few specific types of
vulns with their DAST scanner rather than having to scan the entire
Benchmark each time.

On 12/10/15, 1:43 AM, "Kevin W. Wall" <kevin.w.wall at gmail.com> wrote:

>Was just reading more closely on the wiki page under the 'Test Cases'
>tab how it mentioned that the # of test cases have been reduced
>from over 20k to ~3k (specifically, 2740). The wiki page states:
>    Version 1.0 of the Benchmark was published on April 15, 2015
>    and had 20,983 test cases. On May 23, 2015, version 1.1 of
>    the Benchmark was released. The 1.1 release improves on the
>    previous version by making sure that there are both true
>    positives and false positives in every vulnerability area.
>    Version 1.2beta was released on August 15, 2015.
>    Version 1.2 and forward of the Benchmark is a fully
>    executable web application, which means it is scannable by
>    any kind of vulnerability detection tool. The 1.2beta has
>    been limited to slightly less than 3,000 test cases, to make
>    it easier for DAST tools to scan it (so it doesn't take so
>    long and they don't run out of memory, or blow up the size of
>    their database). The final 1.2 release is expected to be at
>    least 5,000 or possibly 10,000 test cases, after we determine
>    that the popular DAST scanners can handle that size.
>I was going to suggest that perhaps one way to deal with the
>complexity and length run times of DAST would be to all whomever
>is configuring the tests to have a "selector" that just allows one
>or more CWEs to be chosen as the test cases and then only
>deploy those test cases matching the selected CWEs into the
>mix for the DAST testing. (SAST should be fast enough to probably
>run against all test cases.)
>Note that I'm not volunteering to write the code, but one thing is that
>goes back to my 1st question about contributing to the test cases.
>If we are not going to put all 20k of them out there, then it would be
>difficult to tell if some of them are redundant. And if there is a desire
>to put them all out there (which I believe should be the ultimate goal),
>then we need some better way to organize them for people to contribute,
>e.g., making 'cwe##' subdirectories for the test cases in
>    src/main/java/org/owasp/benchmark/testcode
>and organize them that way. (That would also make creating a 'selector'
>a bit easier.)
>Just my $.02,
>Blog: http://off-the-wall-security.blogspot.com/
>NSA: All your crypto bit are belong to us.
>OWASP-benchmark-project mailing list
>OWASP-benchmark-project at lists.owasp.org

More information about the OWASP-benchmark-project mailing list