[Owasp-leaders] OWASP Benchmark project - potential conflict of interest

Konstantinos Papapanagiotou Konstantinos at owasp.org
Tue Dec 1 22:56:21 UTC 2015


Of course it's hard! If it wasn't we'd have tons of "benchmark" programs.
Jeff, I am absolutely certain that you've done a great, objective work with
the benchmark project.
I strongly believe that owasp as a vendor neutral organization should not
make any kind of judgement or provide metrics for comparing vendors. If I
was a competing vendor I'd be seriously considering to withdraw my support
for owasp eg as a corporate sponsor.



On Wednesday, 2 December 2015, Jeff Williams <jeff.williams at owasp.org>
wrote:

> You've mischaracterized the situation here. Gartner has no prayer of
> producing something like this.  It's not like it's hard to verify that the
> test cases are fair and reasonable.
>
> --Jeff
> _____________________________
> From: Tony Turner <tony.turner at owasp.org
> <javascript:_e(%7B%7D,'cvml','tony.turner at owasp.org');>>
> Sent: Tuesday, December 1, 2015 3:05 PM
> Subject: Re: [Owasp-leaders] [Owasp-board] OWASP Benchmark project -
> potential conflict of interest
> To: Konstantinos Papapanagiotou <konstantinos at owasp.org
> <javascript:_e(%7B%7D,'cvml','konstantinos at owasp.org');>>
> Cc: OWASP Foundation Board List <owasp-board at lists.owasp.org
> <javascript:_e(%7B%7D,'cvml','owasp-board at lists.owasp.org');>>, <
> owasp-leaders at lists.owasp.org
> <javascript:_e(%7B%7D,'cvml','owasp-leaders at lists.owasp.org');>>
>
>
> I would oppose setting a precedent that states an OWASP project cannot
> perform such evaluations. We just need to establish some common sense
> guidelines here, like not allowing project leadership to biased by
> employer/partnership relationships or requiring a multi-leader PM
> structure. What I do think makes sense, is for OWASP the organization to
> not rank products. Its a very fine distinction to be sure.
>
> I do NOT have an issue with the benchmark project's core purpose. I DO
> have an issue with it being led by Contrast (or any SAST/DAST/IAST vendor).
> I DO have an issue with their abuse of OWASP brand for marketing purposes.
>
>
> On Tue, Dec 1, 2015 at 1:57 PM, Konstantinos Papapanagiotou <
> Konstantinos at owasp.org
> <javascript:_e(%7B%7D,'cvml','Konstantinos at owasp.org');>> wrote:
>
>> My 2 cents on this:
>>
>> I am quite (and unpleasantly) surprised that we need several threads, a
>> resignation of a volunteer and numerous comments on social media to debate
>> and reach a conclusion on this issue. From where I come from the situation
>> has been quite clear from the beginning. Imagine an antivirus vendor
>> building an av assessment tool that -what a coincidence- ranks them as
>> their product as the best av, donating it to VirusBulletin and then saying
>> that VirusBulletin has concluded that their av is the best. Personally it
>> makes me feel stupid; Contrast's marketing approach undermines my
>> intelligence.
>>
>> The glass is broken. In my opinion we need an official statement saying
>> the "the benchmark" is no longer an OWASP project and that OWASP does not
>> endorse any vendor. In my opinion OWASP should not be in the business of
>> testing and ranking different commercial products. We are not Gartner and
>> we should not become Gartner. I honestly can't see any other way of fixing
>> this apart from removing the project from the OWASP inventory.
>>
>> Kostas
>>
>> On Mon, Nov 30, 2015 at 7:17 PM, psiinon <psiinon at gmail.com
>> <javascript:_e(%7B%7D,'cvml','psiinon at gmail.com');>> wrote:
>>
>>> I'd like to start by saying that I actually _like_ the Benchmark
>>> project.
>>> Myself and other ZAP developers have made some contributions to it, and
>>> we have used (and will continue to use) it to make ZAP better.
>>> I think these sort of testing applications are very valuable to all
>>> security tools, and I'd like to thank Dave and his team for the significant
>>> amount of effort involved in developing and open sourcing it.
>>>
>>> But I dont think it should be an OWASP project.
>>> I do not think that a vendor led project can ever objectively evaluate
>>> competing commercial and open source projects.
>>> I do not think that just saying 'pull requests welcomed' makes a project
>>> vendor neutral.
>>> I do not think that a project as mired in controversy as the Benchmark
>>> project can ever recover to become truly independent.
>>>
>>> I am very disappointed in the Boards handling of this affair.
>>>
>>> Ideally I'd like Dave to understand how much damage this project has
>>> done and to withdraw it as an OWASP project, while still maintaining it as
>>> a very valuable vendor led open source resource.
>>>
>>> Failing that I really hope that the Board comes to its senses and ejects
>>> the Benchmark project before even more damage is done.
>>> At the _very_ least it should flag the project as being 'in dispute' (as
>>> Kevin suggested) while a more detailed evaluation is performed.
>>>
>>> However I'm rapidly loosing loosing faith that the Board will do the
>>> right thing and protect OWASP's image in the way that they should have
>>> already done.
>>> Members - please make your voices heard before more people and projects
>>> leave OWASP.
>>>
>>> Simon
>>>
>>> On Sat, Nov 28, 2015 at 5:14 AM, Jim Manico <jim.manico at owasp.org
>>> <javascript:_e(%7B%7D,'cvml','jim.manico at owasp.org');>> wrote:
>>>
>>>> WAFEC does not "do vendor assessment"; they define a comprehensive
>>>> standard built by many vendors and let the community use that standard to
>>>> measure tools on their own. Just a FYI, I was involved in the early version
>>>> of this project. (Things may have changed since my involvement, I'm sure
>>>> Tony has more details here)
>>>>
>>>> Johanna's comments on this issue lead me to believe that the damage
>>>> done to both OWASP and DHS is even more destructive that I thought. It
>>>> saddens me to see this level of abuse just to sell product.
>>>>
>>>> --
>>>> Jim Manico
>>>> Global Board Member
>>>> OWASP Foundation
>>>> https://www.owasp.org
>>>> Join me in Rome for AppSecEU 2016!
>>>>
>>>> On Nov 28, 2015, at 2:40 AM, Josh Sokol < josh.sokol at owasp.org
>>>> <javascript:_e(%7B%7D,'cvml','josh.sokol at owasp.org');>> wrote:
>>>>
>>>> One of the ideas that Andrew proposed was actually approaching WAFEC to
>>>> learn more about how they do vendor assessment in a neutral way.  It's
>>>> great to hear that we have a resource here already that we can leverage.  I
>>>> wasn't aware of your affiliation.
>>>>
>>>> ~josh
>>>> On Nov 27, 2015 2:47 PM, "Tony Turner" < tony.turner at owasp.org
>>>> <javascript:_e(%7B%7D,'cvml','tony.turner at owasp.org');>> wrote:
>>>>
>>>>> I sincerely hope so. That's not the impression I got from others
>>>>> comments. Personally I haven't used the tool at all, but as I'm the project
>>>>> lead for another product evaluation project (WAFEC) I'm very sensitive to
>>>>> the need of collaboration with many different vendors. There really has to
>>>>> be a very high level (almost paranoid level) transparency with how vendors
>>>>> are approached, worked with, how requirements for evaluation are defined,
>>>>> and how metrics are derived.
>>>>>
>>>>> It appears the project team is attempting to address these last 2
>>>>> somewhat but I'd like to see more specifics, and the lack of information on
>>>>> how they are addressing vendor communication, participation and
>>>>> transparency seems a bit concerning. Lastly, it is my opinion that project
>>>>> leadership should not belong to anyone working for or with a
>>>>> partnership/ownership stake for any vendor being evaluated. I think this is
>>>>> a flawed model and should transition to a vendor neutral party.
>>>>> On Nov 27, 2015 3:16 PM, "Josh Sokol" < josh.sokol at owasp.org
>>>>> <javascript:_e(%7B%7D,'cvml','josh.sokol at owasp.org');>> wrote:
>>>>>
>>>>>> I don't know what qualifies as "significant" in your mind, but my
>>>>>> understanding is that there have been contributions from other vendors:
>>>>>>
>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>>>
>>>>>> Still, Dave would like more, but he can't force them to help.
>>>>>>
>>>>>> ~josh
>>>>>>
>>>>>> On Fri, Nov 27, 2015 at 1:45 PM, Tony Turner <tony.turner at owasp.org
>>>>>> <javascript:_e(%7B%7D,'cvml','tony.turner at owasp.org');>> wrote:
>>>>>>
>>>>>>> While I can appreciate that they started with Contrast, if there
>>>>>>> hasn't been significant effort to include other vendors it's a worthless
>>>>>>> benchmark. It's easy to state you haven't gotten support from other vendors
>>>>>>> and that's fine, but until you do there's really nothing to release. Why
>>>>>>> was it ever upgraded? Talking about the results without an accurate
>>>>>>> comparative analysis is akin to snake oil.
>>>>>>> On Nov 27, 2015 1:49 PM, "Josh Sokol" < josh.sokol at owasp.org
>>>>>>> <javascript:_e(%7B%7D,'cvml','josh.sokol at owasp.org');>> wrote:
>>>>>>>
>>>>>>>> Thank you for the links to those articles.  The first one discusses
>>>>>>>> the strengths and weaknesses of the different methods of evaluating for
>>>>>>>> application vulnerabilities.  The section on the Benchmark seems wholly
>>>>>>>> appropriate to me.  That seems like an excellent description of what the
>>>>>>>> project is designed to do.  I see some metrics in there about which tools
>>>>>>>> are more effective on which types of vulnerabilities, but I don't see him
>>>>>>>> straight up saying "The OWASP Benchmark proves that Contrast is better".
>>>>>>>> This seems like statements made based on some level of testing and
>>>>>>>> research.  Honestly, I don't see any OWASP brand abuse in that article.
>>>>>>>> Whether it's in good taste or not at this stage in the project is certainly
>>>>>>>> debatable, but if you look at the brand usage guidelines (
>>>>>>>> https://www.owasp.org/index.php/Marketing/Resources#tab=BRAND_GUIDELINES),
>>>>>>>> I don't see any violations.  We need to govern to policy here which is why
>>>>>>>> Paul and Noreen are evaluating changes to the guidelines and our
>>>>>>>> enforcement policies to make abuse more difficult.
>>>>>>>>
>>>>>>>> The second article is a competing vendor's reaction to the first.
>>>>>>>> He makes some good points about the issues with Benchmark, but he also says
>>>>>>>> that he hopes that it will be improved over time, and Dave has committed to
>>>>>>>> that.  What I don't see is the vendor saying "...and Veracode has committed
>>>>>>>> resources to help make the Benchmark more accurate across all tool sets".
>>>>>>>> The Benchmark page is pretty clear that it does it's best to provide a
>>>>>>>> benchmark without working exactly like a real-world application.  Maybe
>>>>>>>> some more disclaimer text about where the project is at today would be in
>>>>>>>> order to validate some of Chris' concerns, but I hardly see this as "brand
>>>>>>>> abuse" or a reason to demote the project.
>>>>>>>>
>>>>>>>> Please consider that I have spoken with both Dave and Jeff on this
>>>>>>>> topic and read much of the discussions around it before formulating my
>>>>>>>> opinion.  I doubt that you have done the same so I'm not sure how you can
>>>>>>>> claim that you have researched the issues and all parties involved when you
>>>>>>>> haven't even spoken with the two people whom you are accusing of
>>>>>>>> impropriety.  I have no bias here.  I am simply speaking with the
>>>>>>>> individuals involved, looking at the currently OWASP policies and
>>>>>>>> guidelines, and helping to determine our next steps.
>>>>>>>>
>>>>>>>> ~josh
>>>>>>>>
>>>>>>>> On Fri, Nov 27, 2015 at 12:22 PM, johanna curiel curiel <
>>>>>>>> johanna.curiel at owasp.org> wrote:
>>>>>>>>
>>>>>>>> >>While I agree with you that there has been some brand abuse, it
>>>>>>>> was abuse by Contrast (specifically their marketing department), and not by
>>>>>>>> "these gentlemen" as  you state.
>>>>>>>>
>>>>>>>> Really? ..'some brand abuse'..this is more than brand abuse
>>>>>>>>
>>>>>>>> Josh , please read also the article written by Jeff
>>>>>>>>
>>>>>>>> http://www.darkreading.com/vulnerabilities---threats/why-its-insane-to-trust-static-analysis/a/d-id/1322274?
>>>>>>>>
>>>>>>>>
>>>>>>>> And Veracode's reaction including others in Twitter
>>>>>>>>
>>>>>>>> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>>>>>>>
>>>>>>>> My strong advice is to research the issues and all the parties
>>>>>>>> involved before making statements
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Nov 27, 2015 at 2:07 PM, Josh Sokol <josh.sokol at owasp.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Jim,
>>>>>>>>
>>>>>>>> A concern was expressed to the Board and, frankly, I am insulted by
>>>>>>>> you saying that this was "brushed under the rug".  The Board delegated Matt
>>>>>>>> to talk with Dave and they had a lengthy conversation on the subject.  The
>>>>>>>> Board delegated me to talk with Jeff and we had a lengthy conversation on
>>>>>>>> the subject.  If you do not trust in our abilities to read people, ask the
>>>>>>>> right questions, and provide honest feedback about our conversations, then
>>>>>>>> that's a bigger issue that we should take offline.  After our
>>>>>>>> conversations, we took the time to call a special two-hour session of the
>>>>>>>> Board in order to discuss this subject (and only this subject).  We spoke
>>>>>>>> about all facets of the issue at hand, about the challenges and possible
>>>>>>>> solutions, and concluded on some very concrete next steps.
>>>>>>>>
>>>>>>>> While I agree with you that there has been some brand abuse, it was
>>>>>>>> abuse by Contrast (specifically their marketing department), and not by
>>>>>>>> "these gentlemen" as  you state.  Unless you can point to some sort of
>>>>>>>> evidence showing that Jeff and/or Dave first-hand abused the brand, then I
>>>>>>>> believe that you are speaking with your heart instead of with your head.  I
>>>>>>>> appreciate your passion, but I label this as conspiracy theory because
>>>>>>>> without evidence to support your claims, I cannot accept it as anything
>>>>>>>> other.
>>>>>>>>
>>>>>>>> ~josh
>>>>>>>>
>>>>>>>> On Fri, Nov 27, 2015 at 11:39 AM, Jim Manico <jim.manico at owasp.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Josh,
>>>>>>>>
>>>>>>>> I stand by my comments and perspective, but I'm disheartened that
>>>>>>>> you consider my presentation of facts (and the concerns of many active
>>>>>>>> members of our community) as a "conspiracy theory".
>>>>>>>>
>>>>>>>> In my experience, these kind of comments border on insults and only
>>>>>>>> cause folks to harden their opinions.
>>>>>>>>
>>>>>>>> Once again I feel these gentlemen got away with a kind of brand
>>>>>>>> abuse that is very hurtful to the OWASP community but I am at a loss as to
>>>>>>>> how handle or prevent these kinds of mishaps - especially when board
>>>>>>>> members like yourself seem willing to - from what I see - brush it under
>>>>>>>> the rug.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jim Manico
>>>>>>>> Global Board Member
>>>>>>>> OWASP Foundation
>>>>>>>> https://www.owasp.org
>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>
>>>>>>>> On Nov 27, 2015, at 7:23 PM, Josh Sokol < josh.sokol at owasp.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Admittedly, this was my gut reaction at first as well.  I began
>>>>>>>> linking all of these companies, people, and projects together in my mind
>>>>>>>> (there are some loose links there) and painted a big conspiracy picture
>>>>>>>> similar to what Jim and Dinis have stated.  But, after speaking directly
>>>>>>>> with Jeff, and hearing about the conversation that Dave and Matt had, I've
>>>>>>>> changed my mind.
>>>>>>>>
>>>>>>>> I think it begins with the project itself.  If you aren't sold on
>>>>>>>> the idea of the Benchmark, then you'll never be able to get to the same
>>>>>>>> place.  My original line of thinking was that it was just a bar for vendors
>>>>>>>> to compare their tools against eachother, but that's a bit myopic.  We are
>>>>>>>> in an industry where things evolve very quickly.  As a customer of these
>>>>>>>> tools, I know firsthand that something that a tool does today may not be
>>>>>>>> the case a week from now.  Likewise, new features are being added daily and
>>>>>>>> I need a point-in-time metric to be able to gauge continual effectiveness.
>>>>>>>> Cool, right?  But not a game changer.  The game changer part comes when you
>>>>>>>> realize that by developing and evolving the tests that go into the
>>>>>>>> Benchmark, we are moving the bar higher and higher.  We (OWASP) are
>>>>>>>> effectively setting the standard by which these tools will be compared.  A
>>>>>>>> tool that receives a lower score on the Benchmark today knows exactly what
>>>>>>>> they need to work on in order to pass that test tomorrow and we already
>>>>>>>> have examples of tools that have made improvements because of their
>>>>>>>> Benchmark score (Ask Simon about ZAP's experience with the Benchmark).  I
>>>>>>>> don't think that anyone can argue that the Benchmark project isn't being
>>>>>>>> effective when OWASP's own tools are being driven forward as a result of
>>>>>>>> using it.
>>>>>>>>
>>>>>>>> But, but, but, Dave and Jeff own Aspect and have stock in Contrast
>>>>>>>> and Jeff is the Contrast CTO and Contrast got good scores so it's a
>>>>>>>> conspiracy right?  Is there some code that allows Contrast to use the
>>>>>>>> Benchmark?  Absolutely.  Can you really blame Dave for starting his testing
>>>>>>>> on the effectiveness of the Benchmark with a tool that he owned and is
>>>>>>>> familiar with?  If I were going to start a similar project, there's no
>>>>>>>> question in my mind that I would begin my testing with the tools that I
>>>>>>>> have available to me.  That said, is there code that allows other tools to
>>>>>>>> use the Benchmark?  Absolutely.
>>>>>>>>
>>>>>>>> Regarding "Dave has a history of breaching his duty to be vendor
>>>>>>>> neutral", while I cannot comment on his past actions, I can judge what
>>>>>>>> we've seen recently.  Matt saw a presentation from Dave on the Benchmark at
>>>>>>>> a conference in Chicago.  He said that he felt that the message was
>>>>>>>> appropriate and while IAST tools were mentioned as receiving higher scores,
>>>>>>>> it wasn't a "Contrast is the best" type of message, more of a generality.
>>>>>>>> I saw a very similar (if not the same) talk by Jeff at LASCON 2015 and the
>>>>>>>> message was exactly the same.  I watched the talk expecting some sort of
>>>>>>>> impropriety, but found none.  So, perhaps Dave has abused some privilege
>>>>>>>> granted to him in the past, but what I've seen from him at this point, with
>>>>>>>> respect to the Benchmark, has been appropriate.
>>>>>>>>
>>>>>>>> You have a very good point with respect to the Contrast marketing
>>>>>>>> message around the Benchmark.  It's been completely absurd, over the top,
>>>>>>>> and, in my personal opinion, intolerable.  In fact, I experienced the same
>>>>>>>> thing that you talked about with them at LASCON 2015 where they stood in
>>>>>>>> front of the door of the room Jeff was speaking in and scanned attendees as
>>>>>>>> they went into the talk.  I agree that these types of aggressive marketing
>>>>>>>> tactics cannot be tolerated at OWASP.  In addition, we have seen several
>>>>>>>> marketing messages from them effectively implying that OWASP endorses
>>>>>>>> Contrast.  Clearly this is not OK.  I've spoken with Jeff about it and we
>>>>>>>> agreed that it is not in the Benchmark's best interest to have this
>>>>>>>> aggressive Contrast marketing around it at such an early stage.  He has
>>>>>>>> said that he is not responsible for Contrast's marketing team, but that he
>>>>>>>> would speak with the people who are.  I haven't seen a single message from
>>>>>>>> them since so I'm guessing that he's made good on this promise.  While
>>>>>>>> that's an excellent start, OWASP's takeaway here should be that we need to
>>>>>>>> do a better job with our brand usage guidelines both in terms of the
>>>>>>>> wording and enforcement.  There are many other companies out there that use
>>>>>>>> the OWASP brand and I think that we agree that selective enforcement
>>>>>>>> against Contrast is not the right answer.  Paul and Noreen are actively
>>>>>>>> working on this.  Either way, I think that implying that activities from a
>>>>>>>> vendor's marketing department means that the project is not objective is
>>>>>>>> not inappropriate.  If we feel that the project is not objective, then
>>>>>>>> separate measures need to be taken to drive contribution diversity into
>>>>>>>> it.  That I absolutely agree with and the message from Dave was that he
>>>>>>>> would love to have more contributors to his project.  But, seeing as we
>>>>>>>> cannot force people to work on it, this becomes a matter of "put up or shut
>>>>>>>> up".  The same goes for the experts that you said reviewed the code.  If
>>>>>>>> they feel that it is somehow skewed towards Contrast, they have the power
>>>>>>>> to change that.  Now, if someone tries to participate and Dave tells them
>>>>>>>> "No thanks", then I agree we have a problem, but I don't hear anyone
>>>>>>>> inferring that happened.
>>>>>>>>
>>>>>>>> Please, let's drop the conspiracy theories and focus on the
>>>>>>>> tangible things that we can do to help an OWASP project to be more
>>>>>>>> successful.  Help find more participants to drive diversity, update our
>>>>>>>> brand usage guidelines to prevent abuse, enforce them widely, etc.  Thank
>>>>>>>> you.
>>>>>>>>
>>>>>>>> ~josh
>>>>>>>>
>>>>>>>> On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico <jim.manico at owasp.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Dinis,
>>>>>>>>
>>>>>>>> Like a rare celestial moment when all the planets plus Pluto are
>>>>>>>> aligned, I just read your email on the future of OWASP projects thinking,
>>>>>>>> "Dinis is spot on".
>>>>>>>>
>>>>>>>> Reflecting on projects I manage or work on...
>>>>>>>>
>>>>>>>> The Java Encoder and HTML Sanitizer are likely best moved to Apache
>>>>>>>> now that they have reached a measure of adoption and maturity. Apache would
>>>>>>>> be a much better long term custodian. Perhaps the same for AppSensor, but
>>>>>>>> not my project - just thinking out loud.
>>>>>>>>
>>>>>>>> Other similar defensive projects are still being noodled on, so
>>>>>>>> OWASP is a decent home for these research efforts.
>>>>>>>>
>>>>>>>> The whole tools category is also something to consider. Dependency
>>>>>>>> Check and of course ZAP are some of the best projects that OWASP offers,
>>>>>>>> are they best served where they are today? Both have rich communities of
>>>>>>>> developers but I don't see the foundation doing much to support these
>>>>>>>> efforts.
>>>>>>>>
>>>>>>>> ASVS has the opportunity to effect massive change, I would to love
>>>>>>>> to see major investment and volunteer activity here. Pro tech writer,
>>>>>>>> detailed discourses on each individual requirement, etc. If I was king (and
>>>>>>>> I am not, at all) I would invest in ASVS on a 6 figure scale. (And who
>>>>>>>> started ASVS? Jeff, Dave and Boberski, hat tip to such a marvelous idea).
>>>>>>>> Or maybe moving ASVS to the W3C or IETF would help it grow?
>>>>>>>>
>>>>>>>> The Proactive Controls was a pet project but as we approach 2.0 we
>>>>>>>> have several active/awesome volunteers working on it. We will be making the
>>>>>>>> doc "world editable" to make contributions easy. OWASP seems like a good
>>>>>>>> home for such an awareness doc. Same with T10, especially if community
>>>>>>>> edits are welcome.
>>>>>>>>
>>>>>>>> Anyhow, I'm with you on this Dinis. Once a project starts to reach
>>>>>>>> production quality, spinning off the project as an external project or
>>>>>>>> moving it to a different foundation where managing production software or
>>>>>>>> formal standards is their thing seems realistic.
>>>>>>>>
>>>>>>>> I don't have all the answers here, but your email certainly
>>>>>>>> resonated with me.
>>>>>>>>
>>>>>>>> Aloha,
>>>>>>>> --
>>>>>>>> Jim Manico
>>>>>>>> Global Board Member
>>>>>>>> OWASP Foundation
>>>>>>>> https://www.owasp.org
>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>
>>>>>>>> On Nov 26, 2015, at 11:26 PM, Dinis Cruz < dinis.cruz at owasp.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Jim's reading of this situation is exactly my view on the value of
>>>>>>>> the Contrast tool and how it has been 'pushing' the rules of engagement to
>>>>>>>> an very 'fuzzy' moral/ethical/commercial limit :)
>>>>>>>>
>>>>>>>> As per my last email, a key problem here is the 'perceived
>>>>>>>> expectation' of what is an OWASP project, and how it should be consumed.
>>>>>>>>
>>>>>>>> If you look at the OWASP benchmark as a research project, then the
>>>>>>>> only way it could be making the kind of claims it makes (and have
>>>>>>>> credibility) is if it had evolved from OWASP, with its own (diverse)
>>>>>>>> community
>>>>>>>>
>>>>>>>> On 26 November 2015 at 21:01, Jim Manico <jim.manico at owasp.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I have a different take on this situation but my opinion is the
>>>>>>>> "minority opinion". I will respect the rest of the boards take on this, but
>>>>>>>> here is how I see it.
>>>>>>>>
>>>>>>>> First of all, Jeff has stated that he feels I am attacking him
>>>>>>>> personally from a past personal grudge, and frankly I do not fault him for
>>>>>>>> that perspective since we definitely have history with conflict. So it's
>>>>>>>> fair to take my opinion on this with a grain of salt.
>>>>>>>>
>>>>>>>> I look at this situation from the perspective of a forensic
>>>>>>>> investigator.
>>>>>>>>
>>>>>>>> 1) The Benchmark project had Contrast hooks and only Contrast hooks
>>>>>>>> in it when I reviewed it so this leads me to believe that the project was
>>>>>>>> clearly built with Contrast in mind from the ground up, at least in some
>>>>>>>> way.
>>>>>>>> 3) Dave has a history of breaching his duty to be vendor neutral.
>>>>>>>> He was gifted with a keynote in South Korea a few years ago, and used that
>>>>>>>> opportunity to discuss and pitch Contrast, on stage, during a keynote -
>>>>>>>> with Contrast specific slides. This is just supporting evidence of his
>>>>>>>> intention at OWASP to push Contrast in ways that I think are against the
>>>>>>>> intentions and goals of our foundation.
>>>>>>>> 3) Other experts have reviewed the project and felt that many of
>>>>>>>> the tests were very slanted and almost contrived to support Contrast. I can
>>>>>>>> drag those folks into this conversation, but I do not think that would help
>>>>>>>> in any way. So it's fair to call this point heresy.
>>>>>>>> 4) I do not see this project as revolutionary, at all. Every vendor
>>>>>>>> has their own test suite tuned for their tool. As the benchmark stands
>>>>>>>> today, I see it as just another vendors product-specific benchmark. Mass
>>>>>>>> collaboration from many vendors is not just a "nice to have" but a base
>>>>>>>> requirement to get even close to useful for objective tool measurement.
>>>>>>>> 5) Jeff stating that his Marketing people went over the line is
>>>>>>>> also an admission that - well, they went over the line. By the same token
>>>>>>>> Jeff was in his booth at AppSec USA surrounded by benchmark marketing
>>>>>>>> material, discussing this to prospects and he even asked me and Mr Coates
>>>>>>>> to wade into this debate and support Dave. So to say he was not involved
>>>>>>>> and it was only his marketing people seems a stretch at best.
>>>>>>>> 6) The Contrast marketing team was wandering around the conference
>>>>>>>> zapping folks to get leads, and I asked them to stay in their booth, which
>>>>>>>> is standard conference policy. These folks know better but are again going
>>>>>>>> over the line to sell product at OWASP. There is a better way (like
>>>>>>>> focusing on product capability and language support, have consistent +
>>>>>>>> stellar customer service, have a humble and gracious attitude to all
>>>>>>>> prospects and customers, actively participate in OWASP in a vendor neutral
>>>>>>>> and community supportive way, etc).
>>>>>>>>
>>>>>>>> Please note, I think Contrast is a decent tool, I've offered to
>>>>>>>> resell in the past, and I have recommended it in certain situations - even
>>>>>>>> after this situation arose. I'm stating this out of honestly and desire to
>>>>>>>> put my cards on the table. I truly want Jeff and Dave to be successful.
>>>>>>>> They have dedicated their lives to AppSec and if anyone should win
>>>>>>>> big-time, I hope it's them. I even told Jeff I hope he hits the mother load
>>>>>>>> and donates a little back to OWASP.
>>>>>>>>
>>>>>>>> However, my instinct and evidence tell me that they both went over
>>>>>>>> the line in the use of the OWASP brand to sell product.
>>>>>>>>
>>>>>>>> Now, Jeff makes a good point. We as a board and staff are very poor
>>>>>>>> at enforcing brand management policy and it's not fair to single out
>>>>>>>> Contrast, when many other vendors violate the brand, IMO. Just google OWASP
>>>>>>>> and watch the ads fly that use the OWASP name to sell product.
>>>>>>>>
>>>>>>>> Also, any and every request that was made of Dave to adjust the
>>>>>>>> project for the sake of vendor neutrality was taken very seriously.
>>>>>>>> Regardless of Daves past intentions, he is clearly trying to do the right
>>>>>>>> thing moving forward.
>>>>>>>>
>>>>>>>> I look to "postels principle" in this situation (this is otherwise
>>>>>>>> known as the "robustness principle" and dates back to the creation of TCP)
>>>>>>>> . This is paraphrased as, "Be liberal in what you take from others but be
>>>>>>>> conservative in what you dish out". So I think it's critical that OWASP and
>>>>>>>> any OWASP resource present itself in a strict vendor neutral way. But
>>>>>>>> unless OWASP wants to be much more "even" in the enforcement of brand
>>>>>>>> policy across the board to all violators, we should be fairly lax in the
>>>>>>>> enforcement of these issues from the outside world.
>>>>>>>>
>>>>>>>> I am trying to be objective here. My trigonometry teacher once told
>>>>>>>> me "I'd fail my mother" when I asked him if he would ever fail me (I was an
>>>>>>>> A student). If my mother owned a security company and tried the same stunt,
>>>>>>>> I'd have the same opinions about her actions as well.
>>>>>>>>
>>>>>>>> So what next? Well hello from the other side. I'm going back to
>>>>>>>> listening to Adele's new album where I can
>>>>>>>>
>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151202/78b8dacf/attachment-0001.html>


More information about the OWASP-Leaders mailing list