[Owasp-leaders] A neutral Benchmark

Rory McCune (OWASP) rory.mccune at owasp.org
Sat Nov 28 12:22:39 UTC 2015

Hi All, 


Just to add another 0.02 of your local currency to this, I think if you look at the page that Johanna linked (which is Contrast current marketing position), it’s pretty clear that there’s a problem with how this project is being used by Contrast.


The page has the very strong implication that the US Dept of Homeland Security and OWASP are stating that Contrasts IAST solution is vastly superior to SAST and DAST software.  The page has OWASP logos the title is “OWASP Benchmark project”.  


Now people familiar with exactly how OWASP operates (i.e. anyone can start a project and call it an OWASP project) may say that this is fine as it doesn’t represent an endorsement from OWASP.


However try reading that page as someone who doesn’t know this (i.e. how 99+% of people would)


A person who is unfamiliar with how OWASP operates would, I think, take that page as OWASP the organisation endorsing Contrast’s IAST solution as being better than the alternatives.


“The results of the OWASP Benchmark Project – with its 21,000 test cases – are dramatic” – 


This clearly reads as an endorsement by OWASP of their product, as does 


“The 2015 OWASP Benchmark Project, sponsored by the US Department of Homeland Security (DHS), shows that existing SAST and DAST solutions are leaving businesses vulnerable to attack.”


As has been mentioned elsewhere in this thread, how are all the OWASP sponsor companies who make products in the SAST and DAST world going to react here?  I can’t imagine it will make conversations that OWASP members may have with their employers relating supporting OWASP any easier…


Now I like the idea of a cross-tool comparison, although I think we’d be better working with existing project like sectoolsmarket , but OWASP need to be very careful about allowing companies to give the appearance of an endorsement given our position.







From: owasp-leaders-bounces at lists.owasp.org [mailto:owasp-leaders-bounces at lists.owasp.org] On Behalf Of johanna curiel curiel
Sent: 28 November 2015 01:10
To: Josh Sokol <josh.sokol at owasp.org>
Cc: owasp-leaders at lists.owasp.org; Andre Gironda <andreg+owasp at gmail.com>
Subject: Re: [Owasp-leaders] A neutral Benchmark


I think that OWASP should not be publishing results.


Agree, the person publishing the results is Johanna et al.

Also with a disclaimer: Johanna's opinions do not represent in any way OWASP endorsing or not the tool. This initiative is solely carried on by Johanna etc...


Fact is that due to the dependency of a XML output report of the findings, I can totally assert that this tool cannot compare 1 on 1 any SAST/DAST tools against each other, therefore the claims done by Contrast are totally false:

Contrast dominates SAST & DAST in Speed and Accuracy?


This is so false😂....







On Fri, Nov 27, 2015 at 9:00 PM, Josh Sokol <josh.sokol at owasp.org <mailto:josh.sokol at owasp.org> > wrote:

I really like this idea, Johanna, and it seems inline with Dave's suggestion of having an Advisory Board for the project.  The one thing that I do think that we need to steer clear from, however, is in publishing the results of the tests conducted with the Benchmark.  If others want to test and publish their personal results, that's not something we can stop, but in an effort to be vendor-neutral, I think that OWASP should not be publishing results.



On Fri, Nov 27, 2015 at 4:16 PM, johanna curiel curiel <johanna.curiel at owasp.org <mailto:johanna.curiel at owasp.org> > wrote:

Hi Dave


>>I don¹t have licenses to any of these tools and so far, no one has stepped

up and offered to run any of these tools against the Benchmark.


I think that the Contrast marketing campaign hurt the participation of a promising project before it could take off.


For every specific xml output report , you need to create a parser in order to produce the reports. Without their collaboration or people with licences to test, you won't get their input


As a neutral party with no conflict of interests in this project, I think we can request licenses to these vendors and with the participation of other volunteers that have no commercial ulterior motives to this. I have added Ali Ramzoo, who is also part of the  OWASP Research initiative


We could indeed:

*	Promote that the project is under a neutral research initiative
*	Ask for licenses, 
*	Deploy them in a VM we can all have access to
*	Verify if the tools can produce an XML output report (if not you cannot parse)
*	Discuss with them our findings privately before publishing our findings
*	We have also to be very conscious that if the XML report does not generate all the findings in their tool (as the case of ZAP with Fuzzing) then we need mention this very clear. Otherwise you can hurt the reputation of the tool.


This is how I can help this project and try to create a neutral clean view of a tool which I believe has potential but it needs to shake off all the publicity around Contrast






OWASP-Leaders mailing list
OWASP-Leaders at lists.owasp.org <mailto:OWASP-Leaders at lists.owasp.org> 



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151128/adafde27/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 26583 bytes
Desc: not available
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151128/adafde27/attachment-0001.jpg>

More information about the OWASP-Leaders mailing list