[OWASP-benchmark-project] Project Questions

Michael Coates michael.coates at owasp.org
Thu Oct 8 21:32:59 UTC 2015

Resending in case email list errors caused this email to be lost.

Michael Coates

On Tue, Oct 6, 2015 at 2:58 PM, Michael Coates <michael.coates at owasp.org>

> OWASP Benchmark List,
> I've heard more about this project and am excited about the idea of an
> independent perspective of tool performance. I'm trying to understand a few
> things to better respond to questions from those in the security & OWASP
> community.
> In my mind there are two big areas for consideration in a benchmark
> process.
> 1. Are the benchmarks testing the right areas?
> 2. Is the process for creating the benchmark objective & free from
> conflicts of interest.
> I think as a group OWASP is the right body to align on #1.
> I'd like to ask for some clarifications on item #2. I think it's important
> to avoid actual conflict of interest and also the appearance of conflict of
> interest. The former is obvious why we mustn't have that, the latter is
> critical so others have faith in the tool, process and outputs of the
> process when viewing or hearing about the project.
> 1) Can we clarify whether other individuals have submitted meaningful code
> to the project?
> Observation:
> Nearly all the code commits have come from 1 person (project lead).
> https://github.com/OWASP/Benchmark/graphs/contributors
> 2) Can we clarify the contributions of others and their represented
> organizations?
> Observation:
> The acknowledgements tab listed two developers (Juan Gama & Nick Sanidas)
> both who work at the same company as the project lead. It seems other
> people have submitted some small amounts of material, but overall it seems
> all development has come from the same company.
> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
> 3) Can we clarify in what ways we've mitigated the potential conflict of
> interest and also the appearance of a conflict of interest? This seems like
> the largest blocker for wide spread acceptance of this project and the
> biggest risk.
> Observation:
> The project lead and both of the project developers works for a company
> with very close ties to one of the companies that is evaluated by this
> project. Further, it appears the company is performing very well on the
> project tests.
> 4) If we are going to list tool vendors then I'd recommend listing
> multiple vendors for each category.
> Observation:
> The tools page only lists 1 IAST tool. Since this is the point of the
> potential conflict of interest it is important to list numerous IAST tools.
> https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
> 5) Diverse body with multiple points of view
> Observation:
> There is no indication that multiple stakeholders are present to review
> and decide on the future of this project. If they exist, a new section
> should be added to the project page to raise awareness. If they don't
> exist, we should reevaluate how we are obtaining an independent view of the
> testing process.
> Again, I think the idea of the project is great. From my perspective
> clarifying these questions will help ensure the project is not only
> objective, but also perceived as objective from someone reviewing the
> material. Ultimately this will contribute to the success and growth of the
> project.
> Thanks!
> --
> Michael Coates
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-benchmark-project/attachments/20151008/0cb3b28f/attachment.html>

More information about the OWASP-benchmark-project mailing list