[OWASP-benchmark-project] Proposal for handling time taken for running test cases against DAST scans

Kevin W. Wall kevin.w.wall at gmail.com
Sun Dec 13 03:26:30 UTC 2015


It would seem like one useful extension that could be done would be to add
something to the testcase###.xml files that would distinguish whether the
particular test cases were intended for SAST, DAST, or both. That would
allow us to drastically increase the # of SAST tests while keeping the
longer running DAST tests at a minimum. Alternatively, they could the
separated into distinct sub-directories. That would seem like a useful
ability though because some 'sources' (as in source-to-sink) don't really
apply (as in "not visible to") to (web-based) DAST tools since they only
have a web / HTTP view.

Sent from my Droid; please excuse typos.
On Dec 12, 2015 3:02 PM, "Dave Wichers" <dave.wichers at owasp.org> wrote:

> Kevin,
> The test cases are generators from 3 primary building blocks:
> 1) The source of taint (e.g., request.getParameter(), or headers, or
> cookies, etc.)
> 2) Various kinds of propagation constructs (including none at all).
> 3) A sink (e.g., runtime.exec(), or an HTTP response, or an SQL query).
> Currently, there are about:
> 14 sources
> Over 100 sinks
> ~25 propagators
> Not all of these are compatible with each other. That said, producing ALL
> of the combinations would result in 100K or so test cases, but would
> involve massive redundancy. That¹s why a much smaller subset is reasonable
> to produce because each building block is included between 25 up to 100 or
> more times already. And that¹s with the 2740 test cases.
> I¹m OK with generating a bigger set, and in fact, it should naturally grow
> as we add more building blocks. So I think starting with 2740 dynamic test
> cases is a good balance between having a good / broad set, and the time it
> takes to run a successful scan with agains the Benchmark with Dynamic
> tools.
> -Dave
> P.s. We are working on a UI update for the Benchmark that will organize
> all the test cases into different categories by vulnerability category and
> will even break up the tests within a category as well so we never have
> more than 80 test cases for a single directory in the URL space.  This
> will make it easy for someone to test just one or a few specific types of
> vulns with their DAST scanner rather than having to scan the entire
> Benchmark each time.
> On 12/10/15, 1:43 AM, "Kevin W. Wall" <kevin.w.wall at gmail.com> wrote:
> >Was just reading more closely on the wiki page under the 'Test Cases'
> >tab how it mentioned that the # of test cases have been reduced
> >from over 20k to ~3k (specifically, 2740). The wiki page states:
> >
> >    Version 1.0 of the Benchmark was published on April 15, 2015
> >    and had 20,983 test cases. On May 23, 2015, version 1.1 of
> >    the Benchmark was released. The 1.1 release improves on the
> >    previous version by making sure that there are both true
> >    positives and false positives in every vulnerability area.
> >    Version 1.2beta was released on August 15, 2015.
> >
> >    Version 1.2 and forward of the Benchmark is a fully
> >    executable web application, which means it is scannable by
> >    any kind of vulnerability detection tool. The 1.2beta has
> >    been limited to slightly less than 3,000 test cases, to make
> >    it easier for DAST tools to scan it (so it doesn't take so
> >    long and they don't run out of memory, or blow up the size of
> >    their database). The final 1.2 release is expected to be at
> >    least 5,000 or possibly 10,000 test cases, after we determine
> >    that the popular DAST scanners can handle that size.
> >
> >
> >I was going to suggest that perhaps one way to deal with the
> >complexity and length run times of DAST would be to all whomever
> >is configuring the tests to have a "selector" that just allows one
> >or more CWEs to be chosen as the test cases and then only
> >deploy those test cases matching the selected CWEs into the
> >mix for the DAST testing. (SAST should be fast enough to probably
> >run against all test cases.)
> >
> >Note that I'm not volunteering to write the code, but one thing is that
> >this
> >goes back to my 1st question about contributing to the test cases.
> >
> >If we are not going to put all 20k of them out there, then it would be
> >difficult to tell if some of them are redundant. And if there is a desire
> >to put them all out there (which I believe should be the ultimate goal),
> >then we need some better way to organize them for people to contribute,
> >e.g., making 'cwe##' subdirectories for the test cases in
> >    src/main/java/org/owasp/benchmark/testcode
> >and organize them that way. (That would also make creating a 'selector'
> >a bit easier.)
> >
> >Just my $.02,
> >-kevin
> >--
> >Blog: http://off-the-wall-security.blogspot.com/
> >NSA: All your crypto bit are belong to us.
> >_______________________________________________
> >OWASP-benchmark-project mailing list
> >OWASP-benchmark-project at lists.owasp.org
> >https://lists.owasp.org/mailman/listinfo/owasp-benchmark-project
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-benchmark-project/attachments/20151212/0a285093/attachment.html>

More information about the OWASP-benchmark-project mailing list