[Owasp-leaders] A neutral Benchmark

johanna curiel curiel johanna.curiel at owasp.org
Sat Nov 28 01:10:11 UTC 2015


I think that OWASP should not be publishing results.

Agree, the person publishing the results is Johanna et al.
Also with a disclaimer: Johanna's opinions do not represent in any way
OWASP endorsing or not the tool. This initiative is solely carried on by
Johanna etc...

Fact is that due to the dependency of a XML output report of the findings,
I can totally assert that this tool cannot compare 1 on 1 any SAST/DAST
tools against each other, therefore the claims done by Contrast are totally
false:
Contrast dominates SAST & DAST in Speed and Accuracy?

This is so false😂....

http://www.contrastsecurity.com/owasp-benchmark

[image: Inline image 1]



On Fri, Nov 27, 2015 at 9:00 PM, Josh Sokol <josh.sokol at owasp.org> wrote:

> I really like this idea, Johanna, and it seems inline with Dave's
> suggestion of having an Advisory Board for the project.  The one thing that
> I do think that we need to steer clear from, however, is in publishing the
> results of the tests conducted with the Benchmark.  If others want to test
> and publish their personal results, that's not something we can stop, but
> in an effort to be vendor-neutral, I think that OWASP should not be
> publishing results.
>
> ~josh
>
> On Fri, Nov 27, 2015 at 4:16 PM, johanna curiel curiel <
> johanna.curiel at owasp.org> wrote:
>
>> Hi Dave
>>
>> >>I don¹t have licenses to any of these tools and so far, no one has
>> stepped
>> up and offered to run any of these tools against the Benchmark.
>>
>> I think that the Contrast marketing campaign hurt the participation of a
>> promising project before it could take off.
>>
>> For every specific xml output report , you need to create a parser in
>> order to produce the reports. Without their collaboration or people with
>> licences to test, you won't get their input
>>
>> As a neutral party with no conflict of interests in this project, I think
>> we can request licenses to these vendors and with the participation of
>> other volunteers that have no commercial ulterior motives to this. I have
>> added Ali Ramzoo, who is also part of the  OWASP Research initiative
>>
>> We could indeed:
>>
>>    - Promote that the project is under a neutral research initiative
>>    - Ask for licenses,
>>    - Deploy them in a VM we can all have access to
>>    - Verify if the tools can produce an XML output report (if not you
>>    cannot parse)
>>    - Discuss with them our findings privately before publishing our
>>    findings
>>    - We have also to be very conscious that if the XML report does not
>>    generate all the findings in their tool (as the case of ZAP with Fuzzing)
>>    then we need mention this very clear. Otherwise you can hurt the reputation
>>    of the tool.
>>
>>
>> This is how I can help this project and try to create a neutral clean
>> view of a tool which I believe has potential but it needs to shake off all
>> the publicity around Contrast
>>
>> Regards
>>
>> Johanna
>>
>> _______________________________________________
>> OWASP-Leaders mailing list
>> OWASP-Leaders at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151127/0b7c9bf7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot 2015-11-27 21.06.49.png
Type: image/png
Size: 526209 bytes
Desc: not available
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151127/0b7c9bf7/attachment-0001.png>


More information about the OWASP-Leaders mailing list