[Owasp-leaders] OWASP Benchmark project - potential conflict of interest

Jonathan Carter jonathan.carter at owasp.org
Tue Oct 20 18:51:09 UTC 2015


 I just received the exact same email and was about to forward it out to
the group.

On Tue, Oct 20, 2015 at 10:29 AM, Mario Robles OWASP <mario.robles at owasp.org
> wrote:

> We received this today in the place I work, I think this matter is in the
> way to become a big issue soon and sorry if I’m wrong but it seems that the
> Benchmark project was conceived as a marketing campaign or at least that
> was what my boss said about that from an external user perspective
>
> *From:* Contrast Security [mailto:debbi.funk at contrastsecurity.com
> <debbi.funk at contrastsecurity.com>]
> *Sent:* Tuesday, October 20, 2015 11:53 AM
> *To:*
> *Subject:* [IE] Why HP is talking about Contrast
>
>
> Hello,
>
> Would you like to more accurately and quickly secure your application
> portfolio?
>
> A new* industry benchmark *proves Contrast is nearly 200% more accurate
> than the application security products you’re using today.
>
> Our high accuracy means your application security team:
>
>    - Uses fewer resources tracking down security flaws and false alarms
>    - Can secure 100’s or 1000’s of applications, accurately, continuously
>    and in real-time
>    - Will reduce their reliance on manual processes and Application
>    Security experts
>
> That’s why HP is talking about us
> <http://www.contrastsecurity.com/e1t/c/*M2Cy8DyMS48McWRyp5mQpj0/*W4GNvLw8NRXMRW1q9qyd2_898c0/5/f18dQhb0S65N6XxsJFW11Mcw07fttgcVGJwwC5XYrSXW2-kytn2ZPKd4W1hk5V27Jz2FyW16Ch8T92Hp0HW6NcwJ_3bsfx7W5p1nG95J1pPMW5pzY-Z4PCKr5W92n-f926ld0fW8ZFH319k5Yh0W2Z-NyZ5xFD5lW7yQRss7v473vW4WKjHc1VdD9hW5D1J8j3b7myPN2zyGkcwGSXFN2qhNQCg3P4gW4RMP8R5DmLsxF8NH9wv7JRyVZmDs02w9gRkW7qR_wC5Lpf6bW5Fz_X_72tNrDW45_VMF7DsV_VW5ng2zc5zjV5cN4Gky24hQJs2W92YwD81FS__vW64dfFj5WM_NzW33Z0TC44Q-vmW4v31rv2BsYFjW5-H3Hp2Wk9WlW3kW65C6gl-M9W4t26jY7QBpV5W2PcBMn2VQBWHW6LNxB-7HQl_LW24VLVy6SgHfwW3pY1Qr1T1Fb6W2pRgBM9dtlVSVpm816236nsRW2RlddP8zHZFRW54h94B3k6h0yW7cjcXM5NY_1lW4C5Ptq4rJwxfW3pDngY6qwxnVW6ZL-TF4WxLnNW9c4y506jMYSVN622y9GX3hNL102>
>>
> For your review, I've included our Executive Brief on the *OWASP
> Benchmark Project *(click here to download a PDF
> <http://www.contrastsecurity.com/e1t/c/*M2Cy8DyMS48McWRyp5mQpj0/*W5kCzGc8tLNPSW1SC4PL2Jffg10/5/f18dQhb0SbTY8Y9ZwPW9c3V6v4T_wzgW7dKJ4b63JV-zMscTTsXD6prW39Dr-f1wRVfWMYCsnfcPq_nW4s2wsT2mbdb5W6-0mTk6PkHWqW4cVq-D2-BddNW6VBrnl49tCjvW2_dcyc5kcS0yW3r-xL04BJCbyW51n7Vw4VtZVwW62S2tc3jn_BWW4yx_8p2-J-SLW2z_tPY41Q2SWW3c-v1L1nnQd3VQG_nM1X0z2wW7bntT04dpPS4W6G2l8b5VvJ6fW2BcyM91mGWjzW6bwp1Z3hhcFTW654r0t1qQRGBW24K_493MryvDW62KbGc3MdB1LN2HgHpCQ7-qtN3sM3wLRJcCJW9hPxB42HPDD-W7d04VH2z8Xk7W977qk85kpVCpW2nnKJ03KKhSrW1wDQC53Tj5XrW2BRV4x6SSJNcVw4bXP3kYc04W4p7yDT2zz89mN3P_ThjNM4kVW5_MPTV5_N0cdW63RdN21nPLyCW1Ngx1f51b7nsW4vlVH-8nTV6SW56fLS85bX6zXW42-q_k56n8vGW6q-h61632kjyW2ZkDpK8r0M5zf3JhCkl04>).
> Funded by the US Department of Homeland Security, the Benchmark Project
> lets businesses see how effective application security products are, and
> make better decisions.
>
> I’m interested in hearing your thoughts about the Benchmark Project
> <http://www.contrastsecurity.com/e1t/c/*M2Cy8DyMS48McWRyp5mQpj0/*N8RYT6QXC04dW4w24BY7XRn0m0/5/f18dQhb0S5fs8XJ8n0W9g5p2X2qwv1yW3Dmly531SY_hMf59TdXD6prW39Dr-N8pCDMJW96dt2_51RWPtW6c0f_m5C9dJbW4Lf9WW8V08JHW85klNK8lZbC-W5YtTXR5mNLNsW9dSlS635rgClW94-4T07sLbG-VFCG-j5rC5Q6W3l6cbg2kxxnHMHd1g4z74JcVYSMMp6G7FblW4DFfS36RrfGjW4W79Qj5cdM0qW65vwWS5WCHyjW2V38fs5jL9XHW7mxkV337g5Q8W3s_wq62xWf3mW4LV1FX5W-CFFW6QDGyn6cfDJnW8Ryx0y5w1ds6N977qHDF9k1hW61JrW_3TstfLW8qYdGZ6R55z9W8n4LYb1VtdWZW2kdwHy22TNxnV9Mb4N462nxyW67jG_y8r4CwnVNWh-Y5vMHwGW38HsZ3714h8ZW1nxQBx8CgRXfW5x6rt51PNlDpW23rPnB6TNqNMW1QfSYp6_F-jrW5t-_-b2sc1tgW2_BCqF20xt8R0>.
> Let’s schedule a 15-min briefing and we can walk you through the results. Shoot
> me an email <%20debbi.funk at contrastsecurity.com> and let me know which of
> these times works for a next week: Tuesday 10/27 or Thursday 10/29, 8:00 AM
> to 2:00 PM PT.
>
> If I don’t hear from you in the next couple of days, I’ll give you a call.
>
> Regards,
>
> Debbi
>
> *Debbi Funk <debbi.funk at contrastsecurity.com>* ​| Senior Business
> Development Representative
>
> *Contrast Security, Inc.
> <http://www.contrastsecurity.com/e1t/c/*M2Cy8DyMS48McWRyp5mQpj0/*W38dxPd2prfGpW40tFSb7zcXLM0/5/f18dQhb0S65P6XNnX2V11pKp6bN7PpW5XQgwy303wl9W8SpYzc4Bh8-YW3SjD9l2slc6XW5H6gHR3YQ37kW8gTKKV4DD0RFW3BTMRy2zvjkcV7_mdj2bnQVlW5jc9kb4-m_8_W5NzSV54mJBf-W6mvzR59c47kLF5MmW5c7py1W12ccK91QSfkqV5gpxG1chXfGVDSBdq3D3MKLN4LBcd4b80NxN4jVkM8tkzQqW2bklrG50kkyqW1sYmXv2vWcQlVXKC4D7v-Pt0Vs-Xdw87p0NXW7fcZyB2GvPNWW1jBhq33q9gj-W6d1lMR7j1-rbW8sJvbm3W-TPZW6h8w_n8T10wcW1vb0P2311jR7V6BzSh98t2hTN7w7qjjCb_48W8nxYVt6Hql_KVHHX_F3-ZCpdN53fktnpszmbVm9P9T3JSkl1VrdXDH30_t_nW14mmKv4QZ7tPW5cp-J-4GYVJ9W4_8qM66by6nFW1fPzXY8DhDywW2bDnbW7r9dXsW5QNDSc8KRMVRW52BLpc7vzG_dW2n7-CJ2jGh_XW8sTKmD2-9Sl3W3VC1TF1h8ZPMW4lXb7z3bQ-PNW2d9-s26bWgPcW7t67dh26YbSJN8k1q0SnJD8WW2-24Y18KNHDDV_J5T67ycVZkW39s6bW205fdBW3qRhbJ8WNSRNW6MtKMy8XcC__W6yywPh1F-pLxW8N75274HxVFkW97-D__8JvydBW7svmln3pTkvzTvhhX5LFK36103>*
> 408.529.2448 (c) | @contrastsec
> *The World's Fastest Application Security Software*
>
>
>
> On Oct 7, 2015, at 08:54, johanna curiel curiel <johanna.curiel at owasp.org>
> wrote:
>
> At the moment we did the project review, we observed that the project did
> not have enough testing to be considered in any form as 'ready'  for
> benchmarking, neither that it had yet the community adoption, however
> technically speaking as it has been classified by the leaders, the project
> is at the beta stage.
>
> Indeed , Dave had the push to have the project reviewed but it was never
> clear that later on the project was going to be advertisied this way. That
> all happend after the presentation at Appsec.
>
> I had my concerns regarding how sensitive is the subject of the project
> ,but I think we should allow project leaders to develop their communication
> strategy even if this has conflict of interest. It all depends how they
> behave and how they
>
>
> On Tuesday, October 6, 2015, Michael Coates <michael.coates at owasp.org>
> wrote:
>
>> It's not really that formal to add to the agenda, just a wiki that we add
>> in the text.
>>
>> I think you can safely assume it will get the appropriate discussion.
>>
>> On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com> wrote:
>>
>> Really?? Its not on the agenda yet for the next meeting??
>> How does it get added to the agenda?
>> And that was a formal request if that makes any difference :)
>> I'm all in favour of getting the facts straight before any actions are
>> taken, hence my request for an 'ethical review' or whatever it should be
>> called.
>>
>> Cheers,
>>
>> Simon
>>
>> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates <michael.coates at owasp.org>
>> wrote:
>>
>>> First step is to get all of our information straight so we're clear on
>>> where things are at.
>>>
>>> This was not on the board agenda last meeting and is also not on the
>>> next agenda as of yet (of course it could always be added if needed).
>>>
>>> We are aware that people have raised questions though.   I'm hoping we
>>> can get a clear understanding of all the facts and then discuss if changes
>>> are needed.
>>>
>>>
>>>
>>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com> wrote:
>>>
>>> Hey Michael,
>>>
>>> Is the board going to take any action?
>>> Were there any discussions about this controversy in the board meeting
>>> at AppSec USA?
>>> If not will it be on the agenda for the meeting on October 14th?
>>>
>>> Cheers,
>>>
>>> Simon
>>>
>>>
>>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates <michael.coates at owasp.org
>>> > wrote:
>>>
>>>> Simon
>>>>
>>>> I posted the below message earlier today. At this point my goal is to
>>>> just gain clarity over the current reality and ideally drive to a shared
>>>> state of success. This message doesn't seem to be reflected in the list
>>>> yet. It could be because my membership hasn't been approved or because of
>>>> mail list delays (I miss Google groups). But I think these questions will
>>>> start the conversation.
>>>>
>>>> (This was just me asking questions as a curious Owasp member, not any
>>>> action on behalf of the board)
>>>>
>>>>
>>>>
>>>>
>>>> Begin forwarded message:
>>>>
>>>> *From:* Michael Coates <michael.coates at owasp.org>
>>>> *Date:* October 5, 2015 at 6:20:23 PM PDT
>>>> *To:* owasp-benchmark-project at lists.owasp.org
>>>> *Subject:* *Project Questions*
>>>>
>>>> OWASP Benchmark List,
>>>>
>>>> I've heard more about this project and am excited about the idea of an
>>>> independent perspective of tool performance. I'm trying to understand a few
>>>> things to better respond to questions from those in the security & OWASP
>>>> community.
>>>>
>>>> In my mind there are two big areas for consideration in a benchmark
>>>> process.
>>>> 1. Are the benchmarks testing the right areas?
>>>> 2. Is the process for creating the benchmark objective & free from
>>>> conflicts of interest.
>>>>
>>>> I think as a group OWASP is the right body to align on #1.
>>>>
>>>> I'd like to ask for some clarifications on item #2. I think it's
>>>> important to avoid actual conflict of interest and also the appearance of
>>>> conflict of interest. The former is obvious why we mustn't have that, the
>>>> latter is critical so others have faith in the tool, process and outputs of
>>>> the process when viewing or hearing about the project.
>>>>
>>>>
>>>> 1) Can we clarify whether other individuals have submitted meaningful
>>>> code to the project?
>>>> Observation:
>>>> Nearly all the code commits have come from 1 person (project lead).
>>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>>>
>>>> 2) Can we clarify the contributions of others and their represented
>>>> organizations?
>>>> Observation:
>>>> The acknowledgements tab listed two developers (Juan Gama & Nick
>>>> Sanidas) both who work at the same company as the project lead. It seems
>>>> other people have submitted some small amounts of material, but overall it
>>>> seems all development has come from the same company.
>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>
>>>> 3) Can we clarify in what ways we've mitigated the potential conflict
>>>> of interest and also the appearance of a conflict of interest? This seems
>>>> like the largest blocker for wide spread acceptance of this project and the
>>>> biggest risk.
>>>> Observation:
>>>> The project lead and both of the project developers works for a company
>>>> with very close ties to one of the companies that is evaluated by this
>>>> project. Further, it appears the company is performing very well on the
>>>> project tests.
>>>>
>>>> 4) If we are going to list tool vendors then I'd recommend listing
>>>> multiple vendors for each category.
>>>> Observation:
>>>> The tools page only lists 1 IAST tool. Since this is the point of the
>>>> potential conflict of interest it is important to list numerous IAST tools.
>>>> https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
>>>>
>>>> 5) Diverse body with multiple points of view
>>>> Observation:
>>>> There is no indication that multiple stakeholders are present to review
>>>> and decide on the future of this project. If they exist, a new section
>>>> should be added to the project page to raise awareness. If they don't
>>>> exist, we should reevaluate how we are obtaining an independent view of the
>>>> testing process.
>>>>
>>>>
>>>> Again, I think the idea of the project is great. From my perspective
>>>> clarifying these questions will help ensure the project is not only
>>>> objective, but also perceived as objective from someone reviewing the
>>>> material. Ultimately this will contribute to the success and growth of the
>>>> project.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> --
>>>> Michael Coates
>>>>
>>>>
>>>>
>>>>
>>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com> wrote:
>>>>
>>>> OK, based on the concerns raised so far I think the board should
>>>> initiate a review of the OWASP Benchmark project.
>>>> I'm not raising a formal complaint against it, I'm just requesting a
>>>> review.
>>>> And I dont think it needs a 'standard' project review - Johanna has
>>>> already done a very good job of this.
>>>> Not sure what sort of review you'd call it, I'll leave the naming to
>>>> others :)
>>>>
>>>> I'm concerned that we have an OWASP project lead by a company who has a
>>>> clear commercial stake in the results.
>>>> Bringing more companies on board will help, but I'm still not sure that
>>>> alone will make it independent enough.
>>>> Commercial companies can afford to dedicate staff to improving
>>>> Benchmark so that their products look better.
>>>> Open source projects just cant do that, so we are at a distinct
>>>> disadvantage.
>>>> Should we allow a commercially driven OWASP project who's aim could be
>>>> seen be to promote commercial software?
>>>> If so, what sort of checks and balances does it need?
>>>> Those are the sort of questions I'd like an independent review to look
>>>> at.
>>>>
>>>> I do think there are some immediate steps that could be taken:
>>>>
>>>>    - I'd like to see the Benchmark project page clearly state thats
>>>>    its at a very early stage and that the results are _not_ yet suitable for
>>>>    use in commercial literature.
>>>>    - I'd also like the main companies developing Benchmark to be
>>>>    clearly stated on the main page. If and when other companies get involved
>>>>    then this would actually help the project's claim of vendor independence.
>>>>    - And I'd love to see a respected co-leader added to the project
>>>>    who is not associated with any commercial or open source security tools:)
>>>>
>>>> And we should carry on discussing the project on this list - I think
>>>> such discussions are very healthy, and I'd love to see this project mature
>>>> to a state where it can be a trusted, independent and valued resource.
>>>>
>>>> Cheers,
>>>>
>>>> Simon
>>>>
>>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias <tobias.gondrom at owasp.org>
>>>> wrote:
>>>>
>>>>> @Simon:
>>>>> yes, the leaders list is the place for your discussions for project
>>>>> and chapter leaders
>>>>> @Timo: I like your framing of "Don't ask what OWASP can do for me, ask
>>>>> what I can do for OWASP."
>>>>> That should and is indeed the spirit of OWASP:-)
>>>>> Best regards, Tobias
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>>>
>>>>> I don't know enough about the matter to comment on this case, but I
>>>>> feel that any situation where an OWASP project or any OWASP initiative for
>>>>> that matter, is using OWASP to promote its own business interests should be
>>>>> stopped.  We need to get rid of bad apples in OWASP.
>>>>>
>>>>> OWASP is becoming a brand if you would like to think of it that way
>>>>> and we are going to see many more cases of people trying to use OWASP to
>>>>> spread their business interests. At the end of the day everyone should be
>>>>> acting with an attitude of:"Don't ask what OWASP can do for me, ask what I
>>>>> can do for OWASP?"
>>>>>
>>>>>
>>>>>
>>>>> Regards.
>>>>> Timo
>>>>>
>>>>> On Wed, Sep 30, 2015 at 11:48 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>
>>>>>> So, a load of controversy about OWASP Benchmark on twitter, but no
>>>>>> discussion on the leaders list :(
>>>>>> Is this now the wrong place to discuss OWASP projects??
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 24, 2015 at 10:36 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>>
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I've got some concerns about the OWASP Benchmark project.
>>>>>>>
>>>>>>> I _like_ benchmarks, and I'm very pleased to see an active OWASP
>>>>>>> project focused on delivering one.
>>>>>>> I think the project has some technical limitations, but thats fine
>>>>>>> given the stage the project is at, ie _very_ early.
>>>>>>> I dont think that any firm conclusions should be drawn from it until
>>>>>>> its been significantly enhanced.
>>>>>>>
>>>>>>> My concerns are around the marketing that one of the companies
>>>>>>> sponsoring the Benchmark project has started using.
>>>>>>>
>>>>>>> Here we have a company that leads an OWASP project that just happens
>>>>>>> to show that their offering in this area appears to be _significantly_
>>>>>>> better than any of the competition.
>>>>>>> Their recent press release stresses that its an OWASP project, make
>>>>>>> the most of the fact that the US DHS helped fund it but make no mention of
>>>>>>> their role in developing it.
>>>>>>>
>>>>>>> Regardless of the accuracy of the results, it seems like a huge
>>>>>>> conflict of interest :(
>>>>>>>
>>>>>>> It appears that I'm not the only one with concerns related to the
>>>>>>> project:
>>>>>>>
>>>>>>> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>>>>>>
>>>>>>> What do other people think?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Simon
>>>>>>>
>>>>>>> --
>>>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>>>
>>>>>> _______________________________________________
>>>>>> OWASP-Leaders mailing list
>>>>>> OWASP-Leaders at lists.owasp.org
>>>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OWASP-Leaders mailing listOWASP-Leaders at lists.owasp.orghttps://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>
>>>> _______________________________________________
>>>> OWASP-Leaders mailing list
>>>> OWASP-Leaders at lists.owasp.org
>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>>
>>>>
>>>
>>>
>>> --
>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>
>>>
>>
>>
>> --
>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>
>> _______________________________________________
> OWASP-Leaders mailing list
> OWASP-Leaders at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>
>
>
> _______________________________________________
> OWASP-Leaders mailing list
> OWASP-Leaders at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151020/12acb67a/attachment-0001.html>


More information about the OWASP-Leaders mailing list