[Owasp-board] [Owasp-leaders] OWASP Benchmark project - potential conflict of interest

psiinon psiinon at gmail.com
Thu Nov 26 11:13:19 UTC 2015


Paul, and the rest of the board,

Its been over 2 months since I raised this issue.
Whats happening?
Has the board even discussed it?

Cheers,

Simon


On Tue, Oct 20, 2015 at 10:00 PM, Paul Ritchie <paul.ritchie at owasp.org>
wrote:

> Eoin, Johanna, All:
>
> In an earlier email, Josh Sokol mentioned that he will be speaking in the
> next day or 2 to their CTO, while at LASCON, as a representative of the
> OWASP Board.  Following that feedback, the Board has action to take the
> next steps.
>
> Just an FYI that all comments are recognized and action is being taken.
>
> Paul
>
>
>
> Best Regards, Paul Ritchie
> OWASP Executive Director
> paul.ritchie at owasp.org
>
>
> On Tue, Oct 20, 2015 at 1:54 PM, johanna curiel curiel <
> johanna.curiel at owasp.org> wrote:
>
>> Time for owasp to do a public statement and put a clear story regarding
>> this abusive behavior of Owasp brand
>>
>>
>> On Tuesday, October 20, 2015, Eoin Keary <eoin.keary at owasp.org> wrote:
>>
>>> Folks,
>>>
>>> The project should be immediately shelved it's simply bad form.
>>>
>>> This is damaging to OWASP, the industry and exactly what OWASP is not
>>> about.
>>>
>>> There is a clear conflict of interest and distinct lack of science
>>> behind the claims made by Contrast.
>>>
>>>
>>>
>>>
>>>
>>>
>>> Eoin Keary
>>> OWASP Volunteer
>>> @eoinkeary
>>>
>>>
>>>
>>> On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel <
>>> johanna.curiel at owasp.org> wrote:
>>>
>>> At the moment we did the project review, we observed that the project
>>> did not have enough testing to be considered in any form as 'ready'  for
>>> benchmarking, neither that it had yet the community adoption, however
>>> technically speaking as it has been classified by the leaders, the project
>>> is at the beta stage.
>>>
>>> Indeed , Dave had the push to have the project reviewed but it was never
>>> clear that later on the project was going to be advertisied this way. That
>>> all happend after the presentation at Appsec.
>>>
>>> I had my concerns regarding how sensitive is the subject of the project
>>> ,but I think we should allow project leaders to develop their communication
>>> strategy even if this has conflict of interest. It all depends how they
>>> behave and how they manage this.
>>>
>>>
>>> On Tuesday, October 6, 2015, Michael Coates <michael.coates at owasp.org>
>>> wrote:
>>>
>>>> It's not really that formal to add to the agenda, just a wiki that we
>>>> add in the text.
>>>>
>>>> I think you can safely assume it will get the appropriate discussion.
>>>>
>>>> On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com> wrote:
>>>>
>>>> Really?? Its not on the agenda yet for the next meeting??
>>>> How does it get added to the agenda?
>>>> And that was a formal request if that makes any difference :)
>>>> I'm all in favour of getting the facts straight before any actions are
>>>> taken, hence my request for an 'ethical review' or whatever it should be
>>>> called.
>>>>
>>>> Cheers,
>>>>
>>>> Simon
>>>>
>>>> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates <
>>>> michael.coates at owasp.org> wrote:
>>>>
>>>>> First step is to get all of our information straight so we're clear on
>>>>> where things are at.
>>>>>
>>>>> This was not on the board agenda last meeting and is also not on the
>>>>> next agenda as of yet (of course it could always be added if needed).
>>>>>
>>>>> We are aware that people have raised questions though.   I'm hoping we
>>>>> can get a clear understanding of all the facts and then discuss if changes
>>>>> are needed.
>>>>>
>>>>>
>>>>>
>>>>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>
>>>>> Hey Michael,
>>>>>
>>>>> Is the board going to take any action?
>>>>> Were there any discussions about this controversy in the board meeting
>>>>> at AppSec USA?
>>>>> If not will it be on the agenda for the meeting on October 14th?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates <
>>>>> michael.coates at owasp.org> wrote:
>>>>>
>>>>>> Simon
>>>>>>
>>>>>> I posted the below message earlier today. At this point my goal is to
>>>>>> just gain clarity over the current reality and ideally drive to a shared
>>>>>> state of success. This message doesn't seem to be reflected in the list
>>>>>> yet. It could be because my membership hasn't been approved or because of
>>>>>> mail list delays (I miss Google groups). But I think these questions will
>>>>>> start the conversation.
>>>>>>
>>>>>> (This was just me asking questions as a curious Owasp member, not any
>>>>>> action on behalf of the board)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Begin forwarded message:
>>>>>>
>>>>>> *From:* Michael Coates <michael.coates at owasp.org>
>>>>>> *Date:* October 5, 2015 at 6:20:23 PM PDT
>>>>>> *To:* owasp-benchmark-project at lists.owasp.org
>>>>>> *Subject:* *Project Questions*
>>>>>>
>>>>>> OWASP Benchmark List,
>>>>>>
>>>>>> I've heard more about this project and am excited about the idea of
>>>>>> an independent perspective of tool performance. I'm trying to understand a
>>>>>> few things to better respond to questions from those in the security &
>>>>>> OWASP community.
>>>>>>
>>>>>> In my mind there are two big areas for consideration in a benchmark
>>>>>> process.
>>>>>> 1. Are the benchmarks testing the right areas?
>>>>>> 2. Is the process for creating the benchmark objective & free from
>>>>>> conflicts of interest.
>>>>>>
>>>>>> I think as a group OWASP is the right body to align on #1.
>>>>>>
>>>>>> I'd like to ask for some clarifications on item #2. I think it's
>>>>>> important to avoid actual conflict of interest and also the appearance of
>>>>>> conflict of interest. The former is obvious why we mustn't have that, the
>>>>>> latter is critical so others have faith in the tool, process and outputs of
>>>>>> the process when viewing or hearing about the project.
>>>>>>
>>>>>>
>>>>>> 1) Can we clarify whether other individuals have submitted meaningful
>>>>>> code to the project?
>>>>>> Observation:
>>>>>> Nearly all the code commits have come from 1 person (project lead).
>>>>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>>>>>
>>>>>> 2) Can we clarify the contributions of others and their represented
>>>>>> organizations?
>>>>>> Observation:
>>>>>> The acknowledgements tab listed two developers (Juan Gama & Nick
>>>>>> Sanidas) both who work at the same company as the project lead. It seems
>>>>>> other people have submitted some small amounts of material, but overall it
>>>>>> seems all development has come from the same company.
>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>>>
>>>>>> 3) Can we clarify in what ways we've mitigated the potential conflict
>>>>>> of interest and also the appearance of a conflict of interest? This seems
>>>>>> like the largest blocker for wide spread acceptance of this project and the
>>>>>> biggest risk.
>>>>>> Observation:
>>>>>> The project lead and both of the project developers works for a
>>>>>> company with very close ties to one of the companies that is evaluated by
>>>>>> this project. Further, it appears the company is performing very well on
>>>>>> the project tests.
>>>>>>
>>>>>> 4) If we are going to list tool vendors then I'd recommend listing
>>>>>> multiple vendors for each category.
>>>>>> Observation:
>>>>>> The tools page only lists 1 IAST tool. Since this is the point of the
>>>>>> potential conflict of interest it is important to list numerous IAST tools.
>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
>>>>>>
>>>>>> 5) Diverse body with multiple points of view
>>>>>> Observation:
>>>>>> There is no indication that multiple stakeholders are present to
>>>>>> review and decide on the future of this project. If they exist, a new
>>>>>> section should be added to the project page to raise awareness. If they
>>>>>> don't exist, we should reevaluate how we are obtaining an independent view
>>>>>> of the testing process.
>>>>>>
>>>>>>
>>>>>> Again, I think the idea of the project is great. From my perspective
>>>>>> clarifying these questions will help ensure the project is not only
>>>>>> objective, but also perceived as objective from someone reviewing the
>>>>>> material. Ultimately this will contribute to the success and growth of the
>>>>>> project.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Michael Coates
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>>
>>>>>> OK, based on the concerns raised so far I think the board should
>>>>>> initiate a review of the OWASP Benchmark project.
>>>>>> I'm not raising a formal complaint against it, I'm just requesting a
>>>>>> review.
>>>>>> And I dont think it needs a 'standard' project review - Johanna has
>>>>>> already done a very good job of this.
>>>>>> Not sure what sort of review you'd call it, I'll leave the naming to
>>>>>> others :)
>>>>>>
>>>>>> I'm concerned that we have an OWASP project lead by a company who has
>>>>>> a clear commercial stake in the results.
>>>>>> Bringing more companies on board will help, but I'm still not sure
>>>>>> that alone will make it independent enough.
>>>>>> Commercial companies can afford to dedicate staff to improving
>>>>>> Benchmark so that their products look better.
>>>>>> Open source projects just cant do that, so we are at a distinct
>>>>>> disadvantage.
>>>>>> Should we allow a commercially driven OWASP project who's aim could
>>>>>> be seen be to promote commercial software?
>>>>>> If so, what sort of checks and balances does it need?
>>>>>> Those are the sort of questions I'd like an independent review to
>>>>>> look at.
>>>>>>
>>>>>> I do think there are some immediate steps that could be taken:
>>>>>>
>>>>>>    - I'd like to see the Benchmark project page clearly state thats
>>>>>>    its at a very early stage and that the results are _not_ yet suitable for
>>>>>>    use in commercial literature.
>>>>>>    - I'd also like the main companies developing Benchmark to be
>>>>>>    clearly stated on the main page. If and when other companies get involved
>>>>>>    then this would actually help the project's claim of vendor independence.
>>>>>>    - And I'd love to see a respected co-leader added to the project
>>>>>>    who is not associated with any commercial or open source security tools:)
>>>>>>
>>>>>> And we should carry on discussing the project on this list - I think
>>>>>> such discussions are very healthy, and I'd love to see this project mature
>>>>>> to a state where it can be a trusted, independent and valued resource.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias <tobias.gondrom at owasp.org>
>>>>>> wrote:
>>>>>>
>>>>>>> @Simon:
>>>>>>> yes, the leaders list is the place for your discussions for project
>>>>>>> and chapter leaders
>>>>>>> @Timo: I like your framing of "Don't ask what OWASP can do for me,
>>>>>>> ask what I can do for OWASP."
>>>>>>> That should and is indeed the spirit of OWASP:-)
>>>>>>> Best regards, Tobias
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>>>>>
>>>>>>> I don't know enough about the matter to comment on this case, but I
>>>>>>> feel that any situation where an OWASP project or any OWASP initiative for
>>>>>>> that matter, is using OWASP to promote its own business interests should be
>>>>>>> stopped.  We need to get rid of bad apples in OWASP.
>>>>>>>
>>>>>>> OWASP is becoming a brand if you would like to think of it that way
>>>>>>> and we are going to see many more cases of people trying to use OWASP to
>>>>>>> spread their business interests. At the end of the day everyone should be
>>>>>>> acting with an attitude of:"Don't ask what OWASP can do for me, ask what I
>>>>>>> can do for OWASP?"
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Regards.
>>>>>>> Timo
>>>>>>>
>>>>>>> On Wed, Sep 30, 2015 at 11:48 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>>>
>>>>>>>> So, a load of controversy about OWASP Benchmark on twitter, but no
>>>>>>>> discussion on the leaders list :(
>>>>>>>> Is this now the wrong place to discuss OWASP projects??
>>>>>>>>
>>>>>>>> Simon
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Sep 24, 2015 at 10:36 AM, psiinon <psiinon at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi folks,
>>>>>>>>>
>>>>>>>>> I've got some concerns about the OWASP Benchmark project.
>>>>>>>>>
>>>>>>>>> I _like_ benchmarks, and I'm very pleased to see an active OWASP
>>>>>>>>> project focused on delivering one.
>>>>>>>>> I think the project has some technical limitations, but thats fine
>>>>>>>>> given the stage the project is at, ie _very_ early.
>>>>>>>>> I dont think that any firm conclusions should be drawn from it
>>>>>>>>> until its been significantly enhanced.
>>>>>>>>>
>>>>>>>>> My concerns are around the marketing that one of the companies
>>>>>>>>> sponsoring the Benchmark project has started using.
>>>>>>>>>
>>>>>>>>> Here we have a company that leads an OWASP project that just
>>>>>>>>> happens to show that their offering in this area appears to be
>>>>>>>>> _significantly_ better than any of the competition.
>>>>>>>>> Their recent press release stresses that its an OWASP project,
>>>>>>>>> make the most of the fact that the US DHS helped fund it but make no
>>>>>>>>> mention of their role in developing it.
>>>>>>>>>
>>>>>>>>> Regardless of the accuracy of the results, it seems like a huge
>>>>>>>>> conflict of interest :(
>>>>>>>>>
>>>>>>>>> It appears that I'm not the only one with concerns related to the
>>>>>>>>> project:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>>>>>>>>
>>>>>>>>> What do other people think?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Simon
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OWASP-Leaders mailing list
>>>>>>>> OWASP-Leaders at lists.owasp.org
>>>>>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OWASP-Leaders mailing listOWASP-Leaders at lists.owasp.orghttps://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>>>
>>>>>> _______________________________________________
>>>>>> OWASP-Leaders mailing list
>>>>>> OWASP-Leaders at lists.owasp.org
>>>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>
>>>> _______________________________________________
>>> Owasp-board mailing list
>>> Owasp-board at lists.owasp.org
>>> https://lists.owasp.org/mailman/listinfo/owasp-board
>>>
>>>
>> _______________________________________________
>> Owasp-board mailing list
>> Owasp-board at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-board
>>
>>
>
> _______________________________________________
> Owasp-board mailing list
> Owasp-board at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-board
>
>


-- 
OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20151126/5df0d707/attachment-0001.html>


More information about the Owasp-board mailing list