[Owasp-board] [Owasp-leaders] OWASP Benchmark project - potential conflict of interest

johanna curiel curiel johanna.curiel at owasp.org
Tue Oct 20 20:54:58 UTC 2015


Time for owasp to do a public statement and put a clear story regarding
this abusive behavior of Owasp brand

On Tuesday, October 20, 2015, Eoin Keary <eoin.keary at owasp.org> wrote:

> Folks,
>
> The project should be immediately shelved it's simply bad form.
>
> This is damaging to OWASP, the industry and exactly what OWASP is not
> about.
>
> There is a clear conflict of interest and distinct lack of science behind
> the claims made by Contrast.
>
>
>
>
>
>
> Eoin Keary
> OWASP Volunteer
> @eoinkeary
>
>
>
> On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel <
> johanna.curiel at owasp.org
> <javascript:_e(%7B%7D,'cvml','johanna.curiel at owasp.org');>> wrote:
>
> At the moment we did the project review, we observed that the project did
> not have enough testing to be considered in any form as 'ready'  for
> benchmarking, neither that it had yet the community adoption, however
> technically speaking as it has been classified by the leaders, the project
> is at the beta stage.
>
> Indeed , Dave had the push to have the project reviewed but it was never
> clear that later on the project was going to be advertisied this way. That
> all happend after the presentation at Appsec.
>
> I had my concerns regarding how sensitive is the subject of the project
> ,but I think we should allow project leaders to develop their communication
> strategy even if this has conflict of interest. It all depends how they
> behave and how they manage this.
>
>
> On Tuesday, October 6, 2015, Michael Coates <michael.coates at owasp.org
> <javascript:_e(%7B%7D,'cvml','michael.coates at owasp.org');>> wrote:
>
>> It's not really that formal to add to the agenda, just a wiki that we add
>> in the text.
>>
>> I think you can safely assume it will get the appropriate discussion.
>>
>> On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com> wrote:
>>
>> Really?? Its not on the agenda yet for the next meeting??
>> How does it get added to the agenda?
>> And that was a formal request if that makes any difference :)
>> I'm all in favour of getting the facts straight before any actions are
>> taken, hence my request for an 'ethical review' or whatever it should be
>> called.
>>
>> Cheers,
>>
>> Simon
>>
>> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates <michael.coates at owasp.org>
>> wrote:
>>
>>> First step is to get all of our information straight so we're clear on
>>> where things are at.
>>>
>>> This was not on the board agenda last meeting and is also not on the
>>> next agenda as of yet (of course it could always be added if needed).
>>>
>>> We are aware that people have raised questions though.   I'm hoping we
>>> can get a clear understanding of all the facts and then discuss if changes
>>> are needed.
>>>
>>>
>>>
>>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com> wrote:
>>>
>>> Hey Michael,
>>>
>>> Is the board going to take any action?
>>> Were there any discussions about this controversy in the board meeting
>>> at AppSec USA?
>>> If not will it be on the agenda for the meeting on October 14th?
>>>
>>> Cheers,
>>>
>>> Simon
>>>
>>>
>>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates <michael.coates at owasp.org
>>> > wrote:
>>>
>>>> Simon
>>>>
>>>> I posted the below message earlier today. At this point my goal is to
>>>> just gain clarity over the current reality and ideally drive to a shared
>>>> state of success. This message doesn't seem to be reflected in the list
>>>> yet. It could be because my membership hasn't been approved or because of
>>>> mail list delays (I miss Google groups). But I think these questions will
>>>> start the conversation.
>>>>
>>>> (This was just me asking questions as a curious Owasp member, not any
>>>> action on behalf of the board)
>>>>
>>>>
>>>>
>>>>
>>>> Begin forwarded message:
>>>>
>>>> *From:* Michael Coates <michael.coates at owasp.org>
>>>> *Date:* October 5, 2015 at 6:20:23 PM PDT
>>>> *To:* owasp-benchmark-project at lists.owasp.org
>>>> *Subject:* *Project Questions*
>>>>
>>>> OWASP Benchmark List,
>>>>
>>>> I've heard more about this project and am excited about the idea of an
>>>> independent perspective of tool performance. I'm trying to understand a few
>>>> things to better respond to questions from those in the security & OWASP
>>>> community.
>>>>
>>>> In my mind there are two big areas for consideration in a benchmark
>>>> process.
>>>> 1. Are the benchmarks testing the right areas?
>>>> 2. Is the process for creating the benchmark objective & free from
>>>> conflicts of interest.
>>>>
>>>> I think as a group OWASP is the right body to align on #1.
>>>>
>>>> I'd like to ask for some clarifications on item #2. I think it's
>>>> important to avoid actual conflict of interest and also the appearance of
>>>> conflict of interest. The former is obvious why we mustn't have that, the
>>>> latter is critical so others have faith in the tool, process and outputs of
>>>> the process when viewing or hearing about the project.
>>>>
>>>>
>>>> 1) Can we clarify whether other individuals have submitted meaningful
>>>> code to the project?
>>>> Observation:
>>>> Nearly all the code commits have come from 1 person (project lead).
>>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>>>
>>>> 2) Can we clarify the contributions of others and their represented
>>>> organizations?
>>>> Observation:
>>>> The acknowledgements tab listed two developers (Juan Gama & Nick
>>>> Sanidas) both who work at the same company as the project lead. It seems
>>>> other people have submitted some small amounts of material, but overall it
>>>> seems all development has come from the same company.
>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>
>>>> 3) Can we clarify in what ways we've mitigated the potential conflict
>>>> of interest and also the appearance of a conflict of interest? This seems
>>>> like the largest blocker for wide spread acceptance of this project and the
>>>> biggest risk.
>>>> Observation:
>>>> The project lead and both of the project developers works for a company
>>>> with very close ties to one of the companies that is evaluated by this
>>>> project. Further, it appears the company is performing very well on the
>>>> project tests.
>>>>
>>>> 4) If we are going to list tool vendors then I'd recommend listing
>>>> multiple vendors for each category.
>>>> Observation:
>>>> The tools page only lists 1 IAST tool. Since this is the point of the
>>>> potential conflict of interest it is important to list numerous IAST tools.
>>>> https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
>>>>
>>>> 5) Diverse body with multiple points of view
>>>> Observation:
>>>> There is no indication that multiple stakeholders are present to review
>>>> and decide on the future of this project. If they exist, a new section
>>>> should be added to the project page to raise awareness. If they don't
>>>> exist, we should reevaluate how we are obtaining an independent view of the
>>>> testing process.
>>>>
>>>>
>>>> Again, I think the idea of the project is great. From my perspective
>>>> clarifying these questions will help ensure the project is not only
>>>> objective, but also perceived as objective from someone reviewing the
>>>> material. Ultimately this will contribute to the success and growth of the
>>>> project.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> --
>>>> Michael Coates
>>>>
>>>>
>>>>
>>>>
>>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com> wrote:
>>>>
>>>> OK, based on the concerns raised so far I think the board should
>>>> initiate a review of the OWASP Benchmark project.
>>>> I'm not raising a formal complaint against it, I'm just requesting a
>>>> review.
>>>> And I dont think it needs a 'standard' project review - Johanna has
>>>> already done a very good job of this.
>>>> Not sure what sort of review you'd call it, I'll leave the naming to
>>>> others :)
>>>>
>>>> I'm concerned that we have an OWASP project lead by a company who has a
>>>> clear commercial stake in the results.
>>>> Bringing more companies on board will help, but I'm still not sure that
>>>> alone will make it independent enough.
>>>> Commercial companies can afford to dedicate staff to improving
>>>> Benchmark so that their products look better.
>>>> Open source projects just cant do that, so we are at a distinct
>>>> disadvantage.
>>>> Should we allow a commercially driven OWASP project who's aim could be
>>>> seen be to promote commercial software?
>>>> If so, what sort of checks and balances does it need?
>>>> Those are the sort of questions I'd like an independent review to look
>>>> at.
>>>>
>>>> I do think there are some immediate steps that could be taken:
>>>>
>>>>    - I'd like to see the Benchmark project page clearly state thats
>>>>    its at a very early stage and that the results are _not_ yet suitable for
>>>>    use in commercial literature.
>>>>    - I'd also like the main companies developing Benchmark to be
>>>>    clearly stated on the main page. If and when other companies get involved
>>>>    then this would actually help the project's claim of vendor independence.
>>>>    - And I'd love to see a respected co-leader added to the project
>>>>    who is not associated with any commercial or open source security tools:)
>>>>
>>>> And we should carry on discussing the project on this list - I think
>>>> such discussions are very healthy, and I'd love to see this project mature
>>>> to a state where it can be a trusted, independent and valued resource.
>>>>
>>>> Cheers,
>>>>
>>>> Simon
>>>>
>>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias <tobias.gondrom at owasp.org>
>>>> wrote:
>>>>
>>>>> @Simon:
>>>>> yes, the leaders list is the place for your discussions for project
>>>>> and chapter leaders
>>>>> @Timo: I like your framing of "Don't ask what OWASP can do for me, ask
>>>>> what I can do for OWASP."
>>>>> That should and is indeed the spirit of OWASP:-)
>>>>> Best regards, Tobias
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>>>
>>>>> I don't know enough about the matter to comment on this case, but I
>>>>> feel that any situation where an OWASP project or any OWASP initiative for
>>>>> that matter, is using OWASP to promote its own business interests should be
>>>>> stopped.  We need to get rid of bad apples in OWASP.
>>>>>
>>>>> OWASP is becoming a brand if you would like to think of it that way
>>>>> and we are going to see many more cases of people trying to use OWASP to
>>>>> spread their business interests. At the end of the day everyone should be
>>>>> acting with an attitude of:"Don't ask what OWASP can do for me, ask what I
>>>>> can do for OWASP?"
>>>>>
>>>>>
>>>>>
>>>>> Regards.
>>>>> Timo
>>>>>
>>>>> On Wed, Sep 30, 2015 at 11:48 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>
>>>>>> So, a load of controversy about OWASP Benchmark on twitter, but no
>>>>>> discussion on the leaders list :(
>>>>>> Is this now the wrong place to discuss OWASP projects??
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 24, 2015 at 10:36 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I've got some concerns about the OWASP Benchmark project.
>>>>>>>
>>>>>>> I _like_ benchmarks, and I'm very pleased to see an active OWASP
>>>>>>> project focused on delivering one.
>>>>>>> I think the project has some technical limitations, but thats fine
>>>>>>> given the stage the project is at, ie _very_ early.
>>>>>>> I dont think that any firm conclusions should be drawn from it until
>>>>>>> its been significantly enhanced.
>>>>>>>
>>>>>>> My concerns are around the marketing that one of the companies
>>>>>>> sponsoring the Benchmark project has started using.
>>>>>>>
>>>>>>> Here we have a company that leads an OWASP project that just happens
>>>>>>> to show that their offering in this area appears to be _significantly_
>>>>>>> better than any of the competition.
>>>>>>> Their recent press release stresses that its an OWASP project, make
>>>>>>> the most of the fact that the US DHS helped fund it but make no mention of
>>>>>>> their role in developing it.
>>>>>>>
>>>>>>> Regardless of the accuracy of the results, it seems like a huge
>>>>>>> conflict of interest :(
>>>>>>>
>>>>>>> It appears that I'm not the only one with concerns related to the
>>>>>>> project:
>>>>>>>
>>>>>>> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>>>>>>
>>>>>>> What do other people think?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Simon
>>>>>>>
>>>>>>> --
>>>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>>>
>>>>>> _______________________________________________
>>>>>> OWASP-Leaders mailing list
>>>>>> OWASP-Leaders at lists.owasp.org
>>>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OWASP-Leaders mailing listOWASP-Leaders at lists.owasp.orghttps://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>
>>>> _______________________________________________
>>>> OWASP-Leaders mailing list
>>>> OWASP-Leaders at lists.owasp.org
>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>
>>>>
>>>
>>>
>>> --
>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>
>>>
>>
>>
>> --
>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>
>> _______________________________________________
> Owasp-board mailing list
> Owasp-board at lists.owasp.org
> <javascript:_e(%7B%7D,'cvml','Owasp-board at lists.owasp.org');>
> https://lists.owasp.org/mailman/listinfo/owasp-board
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20151020/1dd2455d/attachment-0001.html>


More information about the Owasp-board mailing list