[Owasp-board] [Owasp-leaders] OWASP Benchmark project - potential conflict of interest

Tobias tobias.gondrom at owasp.org
Thu Nov 26 17:55:50 UTC 2015


There have been several conversations on that matter and a dedicated 
call. Unfortunately for personal reasons I could not attend the last 
call as it was at 04:00am my local time, but all other board members did 
participate.

Could please one of my fellow board members give an update.

Best, Tobias



On 26/11/15 18:04, Timo Goosen wrote:
> I would also like to know the answer to Simon's question. We need to 
> get rid of bad apples in OWASP in my opinion, there are too many 
> people just using the OWASP "name" or "brand" to improve their own 
> financial situation or career.
>
> Regards.
> Timo
>
> On Thu, Nov 26, 2015 at 1:13 PM, psiinon <psiinon at gmail.com 
> <mailto:psiinon at gmail.com>> wrote:
>
>     Paul, and the rest of the board,
>
>     Its been over 2 months since I raised this issue.
>     Whats happening?
>     Has the board even discussed it?
>
>     Cheers,
>
>     Simon
>
>
>     On Tue, Oct 20, 2015 at 10:00 PM, Paul Ritchie
>     <paul.ritchie at owasp.org <mailto:paul.ritchie at owasp.org>> wrote:
>
>         Eoin, Johanna, All:
>
>         In an earlier email, Josh Sokol mentioned that he will be
>         speaking in the next day or 2 to their CTO, while at LASCON,
>         as a representative of the OWASP Board.  Following that
>         feedback, the Board has action to take the next steps.
>
>         Just an FYI that all comments are recognized and action is
>         being taken.
>
>         Paul
>
>
>
>         Best Regards, Paul Ritchie
>         OWASP Executive Director
>         paul.ritchie at owasp.org <mailto:paul.ritchie at owasp.org>
>
>
>         On Tue, Oct 20, 2015 at 1:54 PM, johanna curiel curiel
>         <johanna.curiel at owasp.org <mailto:johanna.curiel at owasp.org>>
>         wrote:
>
>             Time for owasp to do a public statement and put a clear
>             story regarding this abusive behavior of Owasp brand
>
>
>             On Tuesday, October 20, 2015, Eoin Keary
>             <eoin.keary at owasp.org <mailto:eoin.keary at owasp.org>> wrote:
>
>                 Folks,
>
>                 The project should be immediately shelved it's simply
>                 bad form.
>
>                 This is damaging to OWASP, the industry and exactly
>                 what OWASP is not about.
>
>                 There is a clear conflict of interest and distinct
>                 lack of science behind the claims made by Contrast.
>
>
>
>
>
>
>                 Eoin Keary
>                 OWASP Volunteer
>                 @eoinkeary
>
>
>
>                 On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel
>                 <johanna.curiel at owasp.org> wrote:
>
>>                 At the moment we did the project review, we observed
>>                 that the project did not have enough testing to be
>>                 considered in any form as 'ready'  for benchmarking,
>>                 neither that it had yet the community adoption,
>>                 however technically speaking as it has been
>>                 classified by the leaders, the project is at the beta
>>                 stage.
>>
>>                 Indeed , Dave had the push to have the project
>>                 reviewed but it was never clear that later on the
>>                 project was going to be advertisied this way. That
>>                 all happend after the presentation at Appsec.
>>
>>                 I had my concerns regarding how sensitive is the
>>                 subject of the project ,but I think we should allow
>>                 project leaders to develop their communication
>>                 strategy even if this has conflict of interest. It
>>                 all depends how they behave and how they manage this.
>>
>>
>>                 On Tuesday, October 6, 2015, Michael Coates
>>                 <michael.coates at owasp.org> wrote:
>>
>>                     It's not really that formal to add to the agenda,
>>                     just a wiki that we add in the text.
>>
>>                     I think you can safely assume it will get the
>>                     appropriate discussion.
>>
>>                     On Oct 6, 2015, at 7:16 AM, psiinon
>>                     <psiinon at gmail.com> wrote:
>>
>>>                     Really?? Its not on the agenda yet for the next
>>>                     meeting??
>>>                     How does it get added to the agenda?
>>>                     And that was a formal request if that makes any
>>>                     difference :)
>>>                     I'm all in favour of getting the facts straight
>>>                     before any actions are taken, hence my request
>>>                     for an 'ethical review' or whatever it should be
>>>                     called.
>>>
>>>                     Cheers,
>>>
>>>                     Simon
>>>
>>>                     On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates
>>>                     <michael.coates at owasp.org> wrote:
>>>
>>>                         First step is to get all of our information
>>>                         straight so we're clear on where things are at.
>>>
>>>                         This was not on the board agenda last
>>>                         meeting and is also not on the next agenda
>>>                         as of yet (of course it could always be
>>>                         added if needed).
>>>
>>>                         We are aware that people have raised
>>>                         questions though.   I'm hoping we can get a
>>>                         clear understanding of all the facts and
>>>                         then discuss if changes are needed.
>>>
>>>
>>>
>>>                         On Oct 6, 2015, at 1:52 AM, psiinon
>>>                         <psiinon at gmail.com> wrote:
>>>
>>>>                         Hey Michael,
>>>>
>>>>                         Is the board going to take any action?
>>>>                         Were there any discussions about this
>>>>                         controversy in the board meeting at AppSec USA?
>>>>                         If not will it be on the agenda for the
>>>>                         meeting on October 14th?
>>>>
>>>>                         Cheers,
>>>>
>>>>                         Simon
>>>>
>>>>
>>>>                         On Tue, Oct 6, 2015 at 8:25 AM, Michael
>>>>                         Coates <michael.coates at owasp.org> wrote:
>>>>
>>>>                             Simon
>>>>
>>>>                             I posted the below message earlier
>>>>                             today. At this point my goal is to just
>>>>                             gain clarity over the current reality
>>>>                             and ideally drive to a shared state of
>>>>                             success. This message doesn't seem to
>>>>                             be reflected in the list yet. It could
>>>>                             be because my membership hasn't been
>>>>                             approved or because of mail list delays
>>>>                             (I miss Google groups). But I think
>>>>                             these questions will start the
>>>>                             conversation.
>>>>
>>>>                             (This was just me asking questions as a
>>>>                             curious Owasp member, not any action on
>>>>                             behalf of the board)
>>>>
>>>>
>>>>
>>>>
>>>>                             Begin forwarded message:
>>>>
>>>>>                             *From:* Michael Coates
>>>>>                             <michael.coates at owasp.org>
>>>>>                             *Date:* October 5, 2015 at 6:20:23 PM PDT
>>>>>                             *To:*
>>>>>                             owasp-benchmark-project at lists.owasp.org
>>>>>                             *Subject:* *Project Questions*
>>>>>
>>>>>                             OWASP Benchmark List,
>>>>>
>>>>>                             I've heard more about this project and
>>>>>                             am excited about the idea of an
>>>>>                             independent perspective of tool
>>>>>                             performance. I'm trying to understand
>>>>>                             a few things to better respond to
>>>>>                             questions from those in the security &
>>>>>                             OWASP community.
>>>>>
>>>>>                             In my mind there are two big areas for
>>>>>                             consideration in a benchmark process.
>>>>>                             1. Are the benchmarks testing the
>>>>>                             right areas?
>>>>>                             2. Is the process for creating the
>>>>>                             benchmark objective & free from
>>>>>                             conflicts of interest.
>>>>>
>>>>>                             I think as a group OWASP is the right
>>>>>                             body to align on #1.
>>>>>
>>>>>                             I'd like to ask for some
>>>>>                             clarifications on item #2. I think
>>>>>                             it's important to avoid actual
>>>>>                             conflict of interest and also the
>>>>>                             appearance of conflict of interest.
>>>>>                             The former is obvious why we mustn't
>>>>>                             have that, the latter is critical so
>>>>>                             others have faith in the tool, process
>>>>>                             and outputs of the process when
>>>>>                             viewing or hearing about the project.
>>>>>
>>>>>
>>>>>                             1) Can we clarify whether other
>>>>>                             individuals have submitted meaningful
>>>>>                             code to the project?
>>>>>                             Observation:
>>>>>                             Nearly all the code commits have come
>>>>>                             from 1 person (project lead).
>>>>>                             https://github.com/OWASP/Benchmark/graphs/contributors
>>>>>
>>>>>                             2) Can we clarify the contributions of
>>>>>                             others and their represented
>>>>>                             organizations?
>>>>>                             Observation:
>>>>>                             The acknowledgements tab listed two
>>>>>                             developers (Juan Gama & Nick Sanidas)
>>>>>                             both who work at the same company as
>>>>>                             the project lead. It seems other
>>>>>                             people have submitted some small
>>>>>                             amounts of material, but overall it
>>>>>                             seems all development has come from
>>>>>                             the same company.
>>>>>                             https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>>
>>>>>                             3) Can we clarify in what ways we've
>>>>>                             mitigated the potential conflict of
>>>>>                             interest and also the appearance of a
>>>>>                             conflict of interest? This seems like
>>>>>                             the largest blocker for wide spread
>>>>>                             acceptance of this project and the
>>>>>                             biggest risk.
>>>>>                             Observation:
>>>>>                             The project lead and both of the
>>>>>                             project developers works for a company
>>>>>                             with very close ties to one of the
>>>>>                             companies that is evaluated by this
>>>>>                             project. Further, it appears the
>>>>>                             company is performing very well on the
>>>>>                             project tests.
>>>>>
>>>>>                             4) If we are going to list tool
>>>>>                             vendors then I'd recommend listing
>>>>>                             multiple vendors for each category.
>>>>>                             Observation:
>>>>>                             The tools page only lists 1 IAST tool.
>>>>>                             Since this is the point of the
>>>>>                             potential conflict of interest it is
>>>>>                             important to list numerous IAST tools.
>>>>>                             https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
>>>>>
>>>>>                             5) Diverse body with multiple points
>>>>>                             of view
>>>>>                             Observation:
>>>>>                             There is no indication that multiple
>>>>>                             stakeholders are present to review and
>>>>>                             decide on the future of this project.
>>>>>                             If they exist, a new section should be
>>>>>                             added to the project page to raise
>>>>>                             awareness. If they don't exist, we
>>>>>                             should reevaluate how we are obtaining
>>>>>                             an independent view of the testing
>>>>>                             process.
>>>>>
>>>>>
>>>>>                             Again, I think the idea of the project
>>>>>                             is great. From my perspective
>>>>>                             clarifying these questions will help
>>>>>                             ensure the project is not only
>>>>>                             objective, but also perceived as
>>>>>                             objective from someone reviewing the
>>>>>                             material. Ultimately this will
>>>>>                             contribute to the success and growth
>>>>>                             of the project.
>>>>>
>>>>>                             Thanks!
>>>>>
>>>>>
>>>>>                             --
>>>>>                             Michael Coates
>>>>>
>>>>>
>>>>>
>>>>
>>>>                             On Oct 2, 2015, at 1:31 AM, psiinon
>>>>                             <psiinon at gmail.com> wrote:
>>>>
>>>>>                             OK, based on the concerns raised so
>>>>>                             far I think the board should initiate
>>>>>                             a review of the OWASP Benchmark project.
>>>>>                             I'm not raising a formal complaint
>>>>>                             against it, I'm just requesting a review.
>>>>>                             And I dont think it needs a 'standard'
>>>>>                             project review - Johanna has already
>>>>>                             done a very good job of this.
>>>>>                             Not sure what sort of review you'd
>>>>>                             call it, I'll leave the naming to
>>>>>                             others :)
>>>>>
>>>>>                             I'm concerned that we have an OWASP
>>>>>                             project lead by a company who has a
>>>>>                             clear commercial stake in the results.
>>>>>                             Bringing more companies on board will
>>>>>                             help, but I'm still not sure that
>>>>>                             alone will make it independent enough.
>>>>>                             Commercial companies can afford to
>>>>>                             dedicate staff to improving Benchmark
>>>>>                             so that their products look better.
>>>>>                             Open source projects just cant do
>>>>>                             that, so we are at a distinct
>>>>>                             disadvantage.
>>>>>                             Should we allow a commercially driven
>>>>>                             OWASP project who's aim could be seen
>>>>>                             be to promote commercial software?
>>>>>                             If so, what sort of checks and
>>>>>                             balances does it need?
>>>>>                             Those are the sort of questions I'd
>>>>>                             like an independent review to look at.
>>>>>
>>>>>                             I do think there are some immediate
>>>>>                             steps that could be taken:
>>>>>
>>>>>                               * I'd like to see the Benchmark
>>>>>                                 project page clearly state thats
>>>>>                                 its at a very early stage and that
>>>>>                                 the results are _not_ yet suitable
>>>>>                                 for use in commercial literature.
>>>>>                               * I'd also like the main companies
>>>>>                                 developing Benchmark to be clearly
>>>>>                                 stated on the main page. If and
>>>>>                                 when other companies get involved
>>>>>                                 then this would actually help the
>>>>>                                 project's claim of vendor
>>>>>                                 independence.
>>>>>                               * And I'd love to see a respected
>>>>>                                 co-leader added to the project who
>>>>>                                 is not associated with any
>>>>>                                 commercial or open source security
>>>>>                                 tools:)
>>>>>
>>>>>                             And we should carry on discussing the
>>>>>                             project on this list - I think such
>>>>>                             discussions are very healthy, and I'd
>>>>>                             love to see this project mature to a
>>>>>                             state where it can be a trusted,
>>>>>                             independent and valued resource.
>>>>>
>>>>>                             Cheers,
>>>>>
>>>>>                             Simon
>>>>>
>>>>>                             On Thu, Oct 1, 2015 at 7:59 PM, Tobias
>>>>>                             <tobias.gondrom at owasp.org> wrote:
>>>>>
>>>>>                                 @Simon:
>>>>>                                 yes, the leaders list is the place
>>>>>                                 for your discussions for project
>>>>>                                 and chapter leaders
>>>>>                                 @Timo: I like your framing of
>>>>>                                 "Don't ask what OWASP can do for
>>>>>                                 me, ask what I can do for OWASP."
>>>>>                                 That should and is indeed the
>>>>>                                 spirit of OWASP:-)
>>>>>                                 Best regards, Tobias
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                                 On 30/09/15 09:42, Timo Goosen wrote:
>>>>>>                                 I don't know enough about the
>>>>>>                                 matter to comment on this case,
>>>>>>                                 but I feel that any situation
>>>>>>                                 where an OWASP project or any
>>>>>>                                 OWASP initiative for that matter,
>>>>>>                                 is using OWASP to promote its own
>>>>>>                                 business interests should be
>>>>>>                                 stopped.  We need to get rid of
>>>>>>                                 bad apples in OWASP.
>>>>>>
>>>>>>                                 OWASP is becoming a brand if you
>>>>>>                                 would like to think of it that
>>>>>>                                 way and we are going to see many
>>>>>>                                 more cases of people trying to
>>>>>>                                 use OWASP to spread their
>>>>>>                                 business interests. At the end of
>>>>>>                                 the day everyone should be acting
>>>>>>                                 with an attitude of:"Don't ask
>>>>>>                                 what OWASP can do for me, ask
>>>>>>                                 what I can do for OWASP?"
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                 Regards.
>>>>>>                                 Timo
>>>>>>
>>>>>>                                 On Wed, Sep 30, 2015 at 11:48 AM,
>>>>>>                                 psiinon <psiinon at gmail.com> wrote:
>>>>>>
>>>>>>                                     So, a load of controversy
>>>>>>                                     about OWASP Benchmark on
>>>>>>                                     twitter, but no discussion on
>>>>>>                                     the leaders list :(
>>>>>>                                     Is this now the wrong place
>>>>>>                                     to discuss OWASP projects??
>>>>>>
>>>>>>                                     Simon
>>>>>>
>>>>>>
>>>>>>                                     On Thu, Sep 24, 2015 at 10:36
>>>>>>                                     AM, psiinon
>>>>>>                                     <psiinon at gmail.com> wrote:
>>>>>>
>>>>>>                                         Hi folks,
>>>>>>
>>>>>>                                         I've got some concerns
>>>>>>                                         about the OWASP Benchmark
>>>>>>                                         project.
>>>>>>
>>>>>>                                         I _like_ benchmarks, and
>>>>>>                                         I'm very pleased to see
>>>>>>                                         an active OWASP project
>>>>>>                                         focused on delivering one.
>>>>>>                                         I think the project has
>>>>>>                                         some technical
>>>>>>                                         limitations, but thats
>>>>>>                                         fine given the stage the
>>>>>>                                         project is at, ie _very_
>>>>>>                                         early.
>>>>>>                                         I dont think that any
>>>>>>                                         firm conclusions should
>>>>>>                                         be drawn from it until
>>>>>>                                         its been significantly
>>>>>>                                         enhanced.
>>>>>>
>>>>>>                                         My concerns are around
>>>>>>                                         the marketing that one of
>>>>>>                                         the companies sponsoring
>>>>>>                                         the Benchmark project has
>>>>>>                                         started using.
>>>>>>
>>>>>>                                         Here we have a company
>>>>>>                                         that leads an OWASP
>>>>>>                                         project that just happens
>>>>>>                                         to show that their
>>>>>>                                         offering in this area
>>>>>>                                         appears to be
>>>>>>                                         _significantly_ better
>>>>>>                                         than any of the competition.
>>>>>>                                         Their recent press
>>>>>>                                         release stresses that its
>>>>>>                                         an OWASP project, make
>>>>>>                                         the most of the fact that
>>>>>>                                         the US DHS helped fund it
>>>>>>                                         but make no mention of
>>>>>>                                         their role in developing it.
>>>>>>
>>>>>>                                         Regardless of the
>>>>>>                                         accuracy of the results,
>>>>>>                                         it seems like a huge
>>>>>>                                         conflict of interest :(
>>>>>>
>>>>>>                                         It appears that I'm not
>>>>>>                                         the only one with
>>>>>>                                         concerns related to the
>>>>>>                                         project:
>>>>>>
>>>>>>                                         https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>>>>>
>>>>>>                                         What do other people think?
>>>>>>
>>>>>>                                         Cheers,
>>>>>>
>>>>>>                                         Simon
>>>>>>
>>>>>>                                         -- 
>>>>>>                                         OWASP ZAP
>>>>>>                                         <https://www.owasp.org/index.php/ZAP>
>>>>>>                                         Project leader
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                     -- 
>>>>>>                                     OWASP ZAP
>>>>>>                                     <https://www.owasp.org/index.php/ZAP>
>>>>>>                                     Project leader
>>>>>>
>>>>>>                                     _______________________________________________
>>>>>>                                     OWASP-Leaders mailing list
>>>>>>                                     OWASP-Leaders at lists.owasp.org
>>>>>>                                     https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>                                 _______________________________________________
>>>>>>                                 OWASP-Leaders mailing list
>>>>>>                                 OWASP-Leaders at lists.owasp.org
>>>>>>                                 https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>                             -- 
>>>>>                             OWASP ZAP
>>>>>                             <https://www.owasp.org/index.php/ZAP>
>>>>>                             Project leader
>>>>>                             _______________________________________________
>>>>>                             OWASP-Leaders mailing list
>>>>>                             OWASP-Leaders at lists.owasp.org
>>>>>                             https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>
>>>>
>>>>
>>>>
>>>>                         -- 
>>>>                         OWASP ZAP
>>>>                         <https://www.owasp.org/index.php/ZAP>
>>>>                         Project leader
>>>
>>>
>>>
>>>
>>>                     -- 
>>>                     OWASP ZAP <https://www.owasp.org/index.php/ZAP>
>>>                     Project leader
>>
>>                 _______________________________________________
>>                 Owasp-board mailing list
>>                 Owasp-board at lists.owasp.org
>>                 https://lists.owasp.org/mailman/listinfo/owasp-board
>
>
>             _______________________________________________
>             Owasp-board mailing list
>             Owasp-board at lists.owasp.org
>             <mailto:Owasp-board at lists.owasp.org>
>             https://lists.owasp.org/mailman/listinfo/owasp-board
>
>
>
>         _______________________________________________
>         Owasp-board mailing list
>         Owasp-board at lists.owasp.org <mailto:Owasp-board at lists.owasp.org>
>         https://lists.owasp.org/mailman/listinfo/owasp-board
>
>
>
>
>     -- 
>     OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>
>     _______________________________________________
>     Owasp-board mailing list
>     Owasp-board at lists.owasp.org <mailto:Owasp-board at lists.owasp.org>
>     https://lists.owasp.org/mailman/listinfo/owasp-board
>
>
>
>
> _______________________________________________
> Owasp-board mailing list
> Owasp-board at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-board

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20151126/df792ad9/attachment-0001.html>


More information about the Owasp-board mailing list