[Owasp-leaders] A neutral Benchmark

Josh Sokol josh.sokol at owasp.org
Sat Nov 28 16:59:05 UTC 2015


I agree 100%, Rory.  The Board (through Paul and Noreen) are working on
effectively a "cease and desist" for that message specifically.

~josh

On Sat, Nov 28, 2015 at 6:22 AM, Rory McCune (OWASP) <rory.mccune at owasp.org>
wrote:

> Hi All,
>
>
>
> Just to add another 0.02 of your local currency to this, I think if you
> look at the page that Johanna linked (which is Contrast current marketing
> position), it’s pretty clear that there’s a problem with how this project
> is being used by Contrast.
>
>
>
> The page has the very strong implication that the US Dept of Homeland
> Security and OWASP are stating that Contrasts IAST solution is vastly
> superior to SAST and DAST software.  The page has OWASP logos the title is
> “OWASP Benchmark project”.
>
>
>
> Now people familiar with exactly how OWASP operates (i.e. anyone can start
> a project and call it an OWASP project) may say that this is fine as it
> doesn’t represent an endorsement from OWASP.
>
>
>
> However try reading that page as someone who doesn’t know this (i.e. how
> 99+% of people would)
>
>
>
> A person who is unfamiliar with how OWASP operates would, I think, take
> that page as OWASP the organisation endorsing Contrast’s IAST solution as
> being better than the alternatives.
>
>
>
> “*The results of the OWASP Benchmark Project – with its 21,000 test cases
> – are dramatic*” –
>
>
>
> This clearly reads as an endorsement by OWASP of their product, as does
>
>
>
> “*The 2015 OWASP Benchmark Project, sponsored by the US Department of
> Homeland Security (DHS), shows that existing SAST and DAST solutions are
> leaving businesses vulnerable to attack.*”
>
>
>
> As has been mentioned elsewhere in this thread, how are all the OWASP
> sponsor companies who make products in the SAST and DAST world going to
> react here?  I can’t imagine it will make conversations that OWASP members
> may have with their employers relating supporting OWASP any easier…
>
>
>
> Now I like the idea of a cross-tool comparison, although I think we’d be
> better working with existing project like sectoolsmarket , but OWASP need
> to be very careful about allowing companies to give the appearance of an
> endorsement given our position.
>
>
>
> Cheers
>
>
>
> Rory
>
>
>
>
>
> *From:* owasp-leaders-bounces at lists.owasp.org [mailto:
> owasp-leaders-bounces at lists.owasp.org] *On Behalf Of *johanna curiel
> curiel
> *Sent:* 28 November 2015 01:10
> *To:* Josh Sokol <josh.sokol at owasp.org>
> *Cc:* owasp-leaders at lists.owasp.org; Andre Gironda <andreg+owasp at gmail.com
> >
> *Subject:* Re: [Owasp-leaders] A neutral Benchmark
>
>
>
> I think that OWASP should not be publishing results.
>
>
>
> Agree, the person publishing the results is Johanna et al.
>
> Also with a disclaimer: Johanna's opinions do not represent in any way
> OWASP endorsing or not the tool. This initiative is solely carried on by
> Johanna etc...
>
>
>
> Fact is that due to the dependency of a XML output report of the findings,
> I can totally assert that this tool cannot compare 1 on 1 any SAST/DAST
> tools against each other, therefore the claims done by Contrast are totally
> false:
>
> Contrast dominates SAST & DAST in Speed and Accuracy?
>
>
>
> This is so false😂....
>
>
>
> http://www.contrastsecurity.com/owasp-benchmark
>
>
>
> [image: Inline image 1]
>
>
>
>
>
>
>
> On Fri, Nov 27, 2015 at 9:00 PM, Josh Sokol <josh.sokol at owasp.org> wrote:
>
> I really like this idea, Johanna, and it seems inline with Dave's
> suggestion of having an Advisory Board for the project.  The one thing that
> I do think that we need to steer clear from, however, is in publishing the
> results of the tests conducted with the Benchmark.  If others want to test
> and publish their personal results, that's not something we can stop, but
> in an effort to be vendor-neutral, I think that OWASP should not be
> publishing results.
>
> ~josh
>
>
>
> On Fri, Nov 27, 2015 at 4:16 PM, johanna curiel curiel <
> johanna.curiel at owasp.org> wrote:
>
> Hi Dave
>
>
>
> >>I don¹t have licenses to any of these tools and so far, no one has
> stepped
>
> up and offered to run any of these tools against the Benchmark.
>
>
>
> I think that the Contrast marketing campaign hurt the participation of a
> promising project before it could take off.
>
>
>
> For every specific xml output report , you need to create a parser in
> order to produce the reports. Without their collaboration or people with
> licences to test, you won't get their input
>
>
>
> As a neutral party with no conflict of interests in this project, I think
> we can request licenses to these vendors and with the participation of
> other volunteers that have no commercial ulterior motives to this. I have
> added Ali Ramzoo, who is also part of the  OWASP Research initiative
>
>
>
> We could indeed:
>
>    - Promote that the project is under a neutral research initiative
>    - Ask for licenses,
>    - Deploy them in a VM we can all have access to
>    - Verify if the tools can produce an XML output report (if not you
>    cannot parse)
>    - Discuss with them our findings privately before publishing our
>    findings
>    - We have also to be very conscious that if the XML report does not
>    generate all the findings in their tool (as the case of ZAP with Fuzzing)
>    then we need mention this very clear. Otherwise you can hurt the reputation
>    of the tool.
>
>
>
> This is how I can help this project and try to create a neutral clean view
> of a tool which I believe has potential but it needs to shake off all the
> publicity around Contrast
>
>
>
> Regards
>
>
>
> Johanna
>
>
>
> _______________________________________________
> OWASP-Leaders mailing list
> OWASP-Leaders at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151128/fc2b599d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 26583 bytes
Desc: not available
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151128/fc2b599d/attachment-0001.jpg>


More information about the OWASP-Leaders mailing list