[Owasp-leaders] OWASP Benchmark project - potential conflict of interest

Mario Robles mario.robles at owasp.org
Tue Oct 20 18:27:35 UTC 2015

El 20 oct 2015, a las 11:29 a.m., Mario Robles OWASP <mario.robles at owasp.org>

We received this today in the place I work, I think this matter is in the
way to become a big issue soon and sorry if I’m wrong but it seems that the
Benchmark project was conceived as a marketing campaign or at least that
was what my boss said about that from an external user perspective

*From:* Contrast Security [mailto:debbi.funk at contrastsecurity.com
<debbi.funk at contrastsecurity.com>]
*Sent:* Tuesday, October 20, 2015 11:53 AM
*Subject:* [IE] Why HP is talking about Contrast


Would you like to more accurately and quickly secure your application

A new* industry benchmark *proves Contrast is nearly 200% more accurate
than the application security products you’re using today.

Our high accuracy means your application security team:

   - Uses fewer resources tracking down security flaws and false alarms
   - Can secure 100’s or 1000’s of applications, accurately, continuously
   and in real-time
   - Will reduce their reliance on manual processes and Application
   Security experts

That’s why HP is talking about us

For your review, I've included our Executive Brief on the *OWASP Benchmark
Project *(click here to download a PDF
Funded by the US Department of Homeland Security, the Benchmark Project
lets businesses see how effective application security products are, and
make better decisions.

I’m interested in hearing your thoughts about the Benchmark Project
Let’s schedule a 15-min briefing and we can walk you through the results. Shoot
me an email <%20debbi.funk at contrastsecurity.com> and let me know which of
these times works for a next week: Tuesday 10/27 or Thursday 10/29, 8:00 AM
to 2:00 PM PT.

If I don’t hear from you in the next couple of days, I’ll give you a call.



*Debbi Funk <debbi.funk at contrastsecurity.com>* ​| Senior Business
Development Representative

*Contrast Security, Inc.
(c) |
*The World's Fastest Application Security Software*

On Oct 7, 2015, at 08:54, johanna curiel curiel <johanna.curiel at owasp.org>

At the moment we did the project review, we observed that the project did
not have enough testing to be considered in any form as 'ready'  for
benchmarking, neither that it had yet the community adoption, however
technically speaking as it has been classified by the leaders, the project
is at the beta stage.

Indeed , Dave had the push to have the project reviewed but it was never
clear that later on the project was going to be advertisied this way. That
all happend after the presentation at Appsec.

I had my concerns regarding how sensitive is the subject of the project
,but I think we should allow project leaders to develop their communication
strategy even if this has conflict of interest. It all depends how they
behave and how they

On Tuesday, October 6, 2015, Michael Coates <michael.coates at owasp.org
<javascript:_e(%7B%7D,'cvml','michael.coates at owasp.org');>> wrote:

> It's not really that formal to add to the agenda, just a wiki that we add
> in the text.
> I think you can safely assume it will get the appropriate discussion.
> On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com> wrote:
> Really?? Its not on the agenda yet for the next meeting??
> How does it get added to the agenda?
> And that was a formal request if that makes any difference :)
> I'm all in favour of getting the facts straight before any actions are
> taken, hence my request for an 'ethical review' or whatever it should be
> called.
> Cheers,
> Simon
> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates <michael.coates at owasp.org>
> wrote:
>> First step is to get all of our information straight so we're clear on
>> where things are at.
>> This was not on the board agenda last meeting and is also not on the next
>> agenda as of yet (of course it could always be added if needed).
>> We are aware that people have raised questions though.   I'm hoping we
>> can get a clear understanding of all the facts and then discuss if changes
>> are needed.
>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com> wrote:
>> Hey Michael,
>> Is the board going to take any action?
>> Were there any discussions about this controversy in the board meeting at
>> AppSec USA?
>> If not will it be on the agenda for the meeting on October 14th?
>> Cheers,
>> Simon
>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates <michael.coates at owasp.org>
>> wrote:
>>> Simon
>>> I posted the below message earlier today. At this point my goal is to
>>> just gain clarity over the current reality and ideally drive to a shared
>>> state of success. This message doesn't seem to be reflected in the list
>>> yet. It could be because my membership hasn't been approved or because of
>>> mail list delays (I miss Google groups). But I think these questions will
>>> start the conversation.
>>> (This was just me asking questions as a curious Owasp member, not any
>>> action on behalf of the board)
>>> Begin forwarded message:
>>> *From:* Michael Coates <michael.coates at owasp.org>
>>> *Date:* October 5, 2015 at 6:20:23 PM PDT
>>> *To:* owasp-benchmark-project at lists.owasp.org
>>> *Subject:* *Project Questions*
>>> OWASP Benchmark List,
>>> I've heard more about this project and am excited about the idea of an
>>> independent perspective of tool performance. I'm trying to understand a few
>>> things to better respond to questions from those in the security & OWASP
>>> community.
>>> In my mind there are two big areas for consideration in a benchmark
>>> process.
>>> 1. Are the benchmarks testing the right areas?
>>> 2. Is the process for creating the benchmark objective & free from
>>> conflicts of interest.
>>> I think as a group OWASP is the right body to align on #1.
>>> I'd like to ask for some clarifications on item #2. I think it's
>>> important to avoid actual conflict of interest and also the appearance of
>>> conflict of interest. The former is obvious why we mustn't have that, the
>>> latter is critical so others have faith in the tool, process and outputs of
>>> the process when viewing or hearing about the project.
>>> 1) Can we clarify whether other individuals have submitted meaningful
>>> code to the project?
>>> Observation:
>>> Nearly all the code commits have come from 1 person (project lead).
>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>> 2) Can we clarify the contributions of others and their represented
>>> organizations?
>>> Observation:
>>> The acknowledgements tab listed two developers (Juan Gama & Nick
>>> Sanidas) both who work at the same company as the project lead. It seems
>>> other people have submitted some small amounts of material, but overall it
>>> seems all development has come from the same company.
>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>> 3) Can we clarify in what ways we've mitigated the potential conflict of
>>> interest and also the appearance of a conflict of interest? This seems like
>>> the largest blocker for wide spread acceptance of this project and the
>>> biggest risk.
>>> Observation:
>>> The project lead and both of the project developers works for a company
>>> with very close ties to one of the companies that is evaluated by this
>>> project. Further, it appears the company is performing very well on the
>>> project tests.
>>> 4) If we are going to list tool vendors then I'd recommend listing
>>> multiple vendors for each category.
>>> Observation:
>>> The tools page only lists 1 IAST tool. Since this is the point of the
>>> potential conflict of interest it is important to list numerous IAST tools.
>>> https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
>>> 5) Diverse body with multiple points of view
>>> Observation:
>>> There is no indication that multiple stakeholders are present to review
>>> and decide on the future of this project. If they exist, a new section
>>> should be added to the project page to raise awareness. If they don't
>>> exist, we should reevaluate how we are obtaining an independent view of the
>>> testing process.
>>> Again, I think the idea of the project is great. From my perspective
>>> clarifying these questions will help ensure the project is not only
>>> objective, but also perceived as objective from someone reviewing the
>>> material. Ultimately this will contribute to the success and growth of the
>>> project.
>>> Thanks!
>>> --
>>> Michael Coates
>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com> wrote:
>>> OK, based on the concerns raised so far I think the board should
>>> initiate a review of the OWASP Benchmark project.
>>> I'm not raising a formal complaint against it, I'm just requesting a
>>> review.
>>> And I dont think it needs a 'standard' project review - Johanna has
>>> already done a very good job of this.
>>> Not sure what sort of review you'd call it, I'll leave the naming to
>>> others :)
>>> I'm concerned that we have an OWASP project lead by a company who has a
>>> clear commercial stake in the results.
>>> Bringing more companies on board will help, but I'm still not sure that
>>> alone will make it independent enough.
>>> Commercial companies can afford to dedicate staff to improving Benchmark
>>> so that their products look better.
>>> Open source projects just cant do that, so we are at a distinct
>>> disadvantage.
>>> Should we allow a commercially driven OWASP project who's aim could be
>>> seen be to promote commercial software?
>>> If so, what sort of checks and balances does it need?
>>> Those are the sort of questions I'd like an independent review to look
>>> at.
>>> I do think there are some immediate steps that could be taken:
>>>    - I'd like to see the Benchmark project page clearly state thats its
>>>    at a very early stage and that the results are _not_ yet suitable for use
>>>    in commercial literature.
>>>    - I'd also like the main companies developing Benchmark to be
>>>    clearly stated on the main page. If and when other companies get involved
>>>    then this would actually help the project's claim of vendor independence.
>>>    - And I'd love to see a respected co-leader added to the project who
>>>    is not associated with any commercial or open source security tools:)
>>> And we should carry on discussing the project on this list - I think
>>> such discussions are very healthy, and I'd love to see this project mature
>>> to a state where it can be a trusted, independent and valued resource.
>>> Cheers,
>>> Simon
>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias <tobias.gondrom at owasp.org> wrote:
>>>> @Simon:
>>>> yes, the leaders list is the place for your discussions for project and
>>>> chapter leaders
>>>> @Timo: I like your framing of "Don't ask what OWASP can do for me, ask
>>>> what I can do for OWASP."
>>>> That should and is indeed the spirit of OWASP:-)
>>>> Best regards, Tobias
>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>> I don't know enough about the matter to comment on this case, but I
>>>> feel that any situation where an OWASP project or any OWASP initiative for
>>>> that matter, is using OWASP to promote its own business interests should be
>>>> stopped.  We need to get rid of bad apples in OWASP.
>>>> OWASP is becoming a brand if you would like to think of it that way and
>>>> we are going to see many more cases of people trying to use OWASP to spread
>>>> their business interests. At the end of the day everyone should be acting
>>>> with an attitude of:"Don't ask what OWASP can do for me, ask what I can do
>>>> for OWASP?"
>>>> Regards.
>>>> Timo
>>>> On Wed, Sep 30, 2015 at 11:48 AM, psiinon <psiinon at gmail.com> wrote:
>>>>> So, a load of controversy about OWASP Benchmark on twitter, but no
>>>>> discussion on the leaders list :(
>>>>> Is this now the wrong place to discuss OWASP projects??
>>>>> Simon
>>>>> On Thu, Sep 24, 2015 at 10:36 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>> Hi folks,
>>>>>> I've got some concerns about the OWASP Benchmark project.
>>>>>> I _like_ benchmarks, and I'm very pleased to see an active OWASP
>>>>>> project focused on delivering one.
>>>>>> I think the project has some technical limitations, but thats fine
>>>>>> given the stage the project is at, ie _very_ early.
>>>>>> I dont think that any firm conclusions should be drawn from it until
>>>>>> its been significantly enhanced.
>>>>>> My concerns are around the marketing that one of the companies
>>>>>> sponsoring the Benchmark project has started using.
>>>>>> Here we have a company that leads an OWASP project that just happens
>>>>>> to show that their offering in this area appears to be _significantly_
>>>>>> better than any of the competition.
>>>>>> Their recent press release stresses that its an OWASP project, make
>>>>>> the most of the fact that the US DHS helped fund it but make no mention of
>>>>>> their role in developing it.
>>>>>> Regardless of the accuracy of the results, it seems like a huge
>>>>>> conflict of interest :(
>>>>>> It appears that I'm not the only one with concerns related to the
>>>>>> project:
>>>>>> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>>>>> What do other people think?
>>>>>> Cheers,
>>>>>> Simon
>>>>>> --
>>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>> --
>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>> _______________________________________________
>>>>> OWASP-Leaders mailing list
>>>>> OWASP-Leaders at lists.owasp.org
>>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>> _______________________________________________
>>>> OWASP-Leaders mailing listOWASP-Leaders at lists.owasp.orghttps://lists.owasp.org/mailman/listinfo/owasp-leaders
>>> --
>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>> _______________________________________________
>>> OWASP-Leaders mailing list
>>> OWASP-Leaders at lists.owasp.org
>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>> --
>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
> --
> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
> _______________________________________________
OWASP-Leaders mailing list
OWASP-Leaders at lists.owasp.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151020/7eed08ca/attachment-0001.html>

More information about the OWASP-Leaders mailing list