[OWASP-benchmark-project] Benchmark Project Initial Announcement

Dave Wichers dave.wichers at owasp.org
Tue Jul 21 13:45:54 UTC 2015

The following was just sent to the OWASP Leaders list to let everyone know
about this new project:


I’m pleased to announce that an OWASP incubator project I’ve been working
on is ready for broad participation. Aspect Security, Denim Group, Secure
Decisions, NIST, DHS, and various tool vendors have all already contributed
to the project (https://www.owasp.org/index.php/Benchmark) and we are eager
for additional input.

The OWASP Benchmark for Application Security Automation (OWASP Benchmark) is
an open test suite designed to help organizations and practitioners
evaluate the speed, coverage, and accuracy of automated application
security testing tools and services. With widespread misunderstanding of
the specific vulnerabilities automated tools cover, end users are often
left with a false sense of security. The OWASP Benchmark will provide
greater visibility and awareness, empowering users to implement the best
combination of tools and verification procedures to meet their security
needs. To join the discussion or keep up with the latest news, please join
the Benchmark project mailing list at:

Version 1.1 of the test suite, which was released May 23rd, contains over
20,000 Java test cases, with a mix of true vulnerabilities and false
positives designed to carefully measure a tool’s accuracy. The following
vulnerability categories are covered in this release:

   - Command Injection
   - Insecure Cookies
   - LDAP Injection
   - Path Traversal
   - SQL Injection
   - Trust Boundary Violations
   - Use of Weak Encryption Algorithms
   - Use of Weak Hashing Algorithms
   - Weak Randomness
   - XPath Injection
   - XSS (Cross-Site Scripting)

The Benchmark scorecard generator was just released on July 10th. It
provides the capability to import analysis results against the Benchmark
from many existing Static Application Security Testing (SAST) tools, and
generate a scorecard describing how those tools fared against the
Benchmark. The Benchmark project can currently generate scorecards for the
following SAST tools:

   - Coverity Code Advisor (On-Demand and stand-alone versions – license
   - Findbugs (open source)
   - FindBugs with the FindSecurityBugs plugin (open source)
   - HP Fortify (On-Demand and stand-alone versions – license required)
   - IBM AppScan Source (license required)
   - Parasoft Jtest (license required)
   - PMD (open source)
   - SonarQube (open source)
   - Veracode SAST (license required)

In a future release, hopefully within 30 days, the Benchmark test cases
will be fully runnable and exploitable, supporting the evaluation of
Dynamic Application Security Testing (DAST) tools, and other application
security tools that require a running application, including Interactive
Application Security Testing (IAST) tools, Web Application Firewalls (WAF),
and Runtime Application Self-Protection (RASP) technology.

The Benchmark has already revealed several big surprises and is pointing
the way towards new strategies for effective application security
automation. Per Dan Cornell, Principal of the Denim Group, "Security
testing tools are an integral part of any application security program. The
OWASP Benchmark is a great resource for organizations looking to evaluate
the effectiveness of security testing tools for their particular

We encourage vendors, open source tools, and end users to verify their
application security tools against the Benchmark and contribute their
results to the project. Check out the project here:
https://www.owasp.org/index.php/Benchmark, and the results for our initial
set of open source tools here:

I will also be talking about the project at AppSec USA 2015, so please come
to my talk to learn more about the project.


Dave Wichers

OWASP Benchmark Project Lead
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-benchmark-project/attachments/20150721/49601cd7/attachment.html>

More information about the OWASP-benchmark-project mailing list