[Owasp-leaders] [Owasp-board] OWASP Benchmark project - potential conflict of interest

Mon Nov 30 17:27:51 UTC 2015

> At the _very_ least it should flag the project as being 'in dispute' (as Kevin
> suggested) while a more detailed evaluation is performed.
+1 

The conflict is clear scrolling through all those E-Mails, Blogs, etc. Until it's sorted out we need a clarification visible to everyone, the Benchmark projects status is heavily discussed within the Community at the Moment. 

> -----Ursprüngliche Nachricht-----
> Von: owasp-leaders-bounces at lists.owasp.org [mailto:owasp-leaders-
> bounces at lists.owasp.org] Im Auftrag von psiinon
> Gesendet: Montag, 30. November 2015 18:18
> An: Jim Manico
> Cc: OWASP Foundation Board List; owasp-leaders at lists.owasp.org
> Betreff: Re: [Owasp-leaders] [Owasp-board] OWASP Benchmark project -
> potential conflict of interest [ Z1 UNGESICHERT ]
> 
> I'd like to start by saying that I actually _like_ the Benchmark project.
> Myself and other ZAP developers have made some contributions to it, and
> we have used (and will continue to use) it to make ZAP better.
> I think these sort of testing applications are very valuable to all security tools,
> and I'd like to thank Dave and his team for the significant amount of effort
> involved in developing and open sourcing it.
> 
> But I dont think it should be an OWASP project.
> I do not think that a vendor led project can ever objectively evaluate
> competing commercial and open source projects.
> I do not think that just saying 'pull requests welcomed' makes a project
> vendor neutral.
> I do not think that a project as mired in controversy as the Benchmark project
> can ever recover to become truly independent.
> 
> I am very disappointed in the Boards handling of this affair.
> 
> Ideally I'd like Dave to understand how much damage this project has done
> and to withdraw it as an OWASP project, while still maintaining it as a very
> valuable vendor led open source resource.
> 
> Failing that I really hope that the Board comes to its senses and ejects the
> Benchmark project before even more damage is done.
> At the _very_ least it should flag the project as being 'in dispute' (as Kevin
> suggested) while a more detailed evaluation is performed.
> 
> However I'm rapidly loosing loosing faith that the Board will do the right thing
> and protect OWASP's image in the way that they should have already done.
> Members - please make your voices heard before more people and projects
> leave OWASP.
> 
> Simon
> 
> 
> On Sat, Nov 28, 2015 at 5:14 AM, Jim Manico <jim.manico at owasp.org>
> wrote:
> 
> 
> 	WAFEC does not "do vendor assessment"; they define a
> comprehensive standard built by many vendors and let the community use
> that standard to measure tools on their own. Just a FYI, I was involved in the
> early version of this project. (Things may have changed since my
> involvement, I'm sure Tony has more details here)
> 
> 	Johanna's comments on this issue lead me to believe that the
> damage done to both OWASP and DHS is even more destructive that I
> thought. It saddens me to see this level of abuse just to sell product.
> 
> 
> 	--
> 	Jim Manico
> 	Global Board Member
> 	OWASP Foundation
> 	https://www.owasp.org <https://www.owasp.org/>
> 	Join me in Rome for AppSecEU 2016!
> 
> 	On Nov 28, 2015, at 2:40 AM, Josh Sokol <josh.sokol at owasp.org>
> wrote:
> 
> 
> 
> 		One of the ideas that Andrew proposed was actually
> approaching WAFEC to learn more about how they do vendor assessment in
> a neutral way.  It's great to hear that we have a resource here already that
> we can leverage.  I wasn't aware of your affiliation.
> 
> 		~josh
> 
> 		On Nov 27, 2015 2:47 PM, "Tony Turner"
> <tony.turner at owasp.org> wrote:
> 
> 
> 			I sincerely hope so. That's not the impression I got
> from others comments. Personally I haven't used the tool at all, but as I'm
> the project lead for another product evaluation project (WAFEC) I'm very
> sensitive to the need of collaboration with many different vendors. There
> really has to be a very high level (almost paranoid level) transparency with
> how vendors are approached, worked with, how requirements for
> evaluation are defined, and how metrics are derived.
> 
> 			It appears the project team is attempting to address
> these last 2 somewhat but I'd like to see more specifics, and the lack of
> information on how they are addressing vendor communication,
> participation and transparency seems a bit concerning. Lastly, it is my opinion
> that project leadership should not belong to anyone working for or with a
> partnership/ownership stake for any vendor being evaluated. I think this is a
> flawed model and should transition to a vendor neutral party.
> 
> 			On Nov 27, 2015 3:16 PM, "Josh Sokol"
> <josh.sokol at owasp.org> wrote:
> 
> 
> 				I don't know what qualifies as "significant" in
> your mind, but my understanding is that there have been contributions from
> other vendors:
> 
> 
> 	https://www.owasp.org/index.php/Benchmark#tab=Acknowledgem
> ents
> 
> 
> 				Still, Dave would like more, but he can't force
> them to help.
> 
> 
> 				~josh
> 
> 
> 				On Fri, Nov 27, 2015 at 1:45 PM, Tony Turner
> <tony.turner at owasp.org> wrote:
> 
> 
> 					While I can appreciate that they
> started with Contrast, if there hasn't been significant effort to include other
> vendors it's a worthless benchmark. It's easy to state you haven't gotten
> support from other vendors and that's fine, but until you do there's really
> nothing to release. Why was it ever upgraded? Talking about the results
> without an accurate comparative analysis is akin to snake oil.
> 
> 					On Nov 27, 2015 1:49 PM, "Josh Sokol"
> <josh.sokol at owasp.org> wrote:
> 
> 
> 						Thank you for the links to
> those articles.  The first one discusses the strengths and weaknesses of the
> different methods of evaluating for application vulnerabilities.  The section
> on the Benchmark seems wholly appropriate to me.  That seems like an
> excellent description of what the project is designed to do.  I see some
> metrics in there about which tools are more effective on which types of
> vulnerabilities, but I don't see him straight up saying "The OWASP Benchmark
> proves that Contrast is better".  This seems like statements made based on
> some level of testing and research.  Honestly, I don't see any OWASP brand
> abuse in that article.  Whether it's in good taste or not at this stage in the
> project is certainly debatable, but if you look at the brand usage guidelines
> (https://www.owasp.org/index.php/Marketing/Resources#tab=BRAND_GU
> IDELINES), I don't see any violations.  We need to govern to policy here which
> is why Paul and Noreen are evaluating changes to the guidelines and our
> enforcement policies to make abuse more difficult.
> 
> 
> 						The second article is a
> competing vendor's reaction to the first.  He makes some good points about
> the issues with Benchmark, but he also says that he hopes that it will be
> improved over time, and Dave has committed to that.  What I don't see is the
> vendor saying "...and Veracode has committed resources to help make the
> Benchmark more accurate across all tool sets".  The Benchmark page is pretty
> clear that it does it's best to provide a benchmark without working exactly
> like a real-world application.  Maybe some more disclaimer text about where
> the project is at today would be in order to validate some of Chris' concerns,
> but I hardly see this as "brand abuse" or a reason to demote the project.
> 
> 
> 						Please consider that I have
> spoken with both Dave and Jeff on this topic and read much of the
> discussions around it before formulating my opinion.  I doubt that you have
> done the same so I'm not sure how you can claim that you have researched
> the issues and all parties involved when you haven't even spoken with the
> two people whom you are accusing of impropriety.  I have no bias here.  I am
> simply speaking with the individuals involved, looking at the currently OWASP
> policies and guidelines, and helping to determine our next steps.
> 
> 
> 						~josh
> 
> 
> 						On Fri, Nov 27, 2015 at 12:22
> PM, johanna curiel curiel <johanna.curiel at owasp.org> wrote:
> 
> 
> 							>>While I agree with
> you that there has been some brand abuse, it was abuse by Contrast
> (specifically their marketing department), and not by "these gentlemen" as
> you state.
> 
> 							Really? ..'some brand
> abuse'..this is more than brand abuse
> 
> 
> 
> 							Josh , please read also
> the article written by Jeff
> 
> 	http://www.darkreading.com/vulnerabilities---threats/why-its-
> insane-to-trust-static-analysis/a/d-id/1322274?
> 
> 
> 
> 							And Veracode's
> reaction including others in Twitter
> 
> 	https://www.veracode.com/blog/2015/09/no-one-technology-
> silver-bullet
> 
> 
> 							My strong advice is to
> research the issues and all the parties involved before making statements
> 
> 
> 
> 
> 							On Fri, Nov 27, 2015 at
> 2:07 PM, Josh Sokol <josh.sokol at owasp.org> wrote:
> 
> 
> 								Jim,
> 
> 
> 								A concern was
> expressed to the Board and, frankly, I am insulted by you saying that this was
> "brushed under the rug".  The Board delegated Matt to talk with Dave and
> they had a lengthy conversation on the subject.  The Board delegated me to
> talk with Jeff and we had a lengthy conversation on the subject.  If you do
> not trust in our abilities to read people, ask the right questions, and provide
> honest feedback about our conversations, then that's a bigger issue that we
> should take offline.  After our conversations, we took the time to call a
> special two-hour session of the Board in order to discuss this subject (and
> only this subject).  We spoke about all facets of the issue at hand, about the
> challenges and possible solutions, and concluded on some very concrete
> next steps.
> 
> 								While I agree
> with you that there has been some brand abuse, it was abuse by Contrast
> (specifically their marketing department), and not by "these gentlemen" as
> you state.  Unless you can point to some sort of evidence showing that Jeff
> and/or Dave first-hand abused the brand, then I believe that you are
> speaking with your heart instead of with your head.  I appreciate your
> passion, but I label this as conspiracy theory because without evidence to
> support your claims, I cannot accept it as anything other.
> 
> 
> 								~josh
> 
> 
> 								On Fri, Nov 27,
> 2015 at 11:39 AM, Jim Manico <jim.manico at owasp.org> wrote:
> 
> 
> 									Josh,
> 
> 									I stand
> by my comments and perspective, but I'm disheartened that you consider
> my presentation of facts (and the concerns of many active members of our
> community) as a "conspiracy theory".
> 
> 									In my
> experience, these kind of comments border on insults and only cause folks
> to harden their opinions.
> 
> 									Once
> again I feel these gentlemen got away with a kind of brand abuse that is very
> hurtful to the OWASP community but I am at a loss as to how handle or
> prevent these kinds of mishaps - especially when board members like
> yourself seem willing to - from what I see - brush it under the rug.
> 
> 
> 									--
> 									Jim
> Manico
> 									Global
> Board Member
> 									OWASP
> Foundation
> 
> 	https://www.owasp.org <https://www.owasp.org/>
> 									Join me
> in Rome for AppSecEU 2016!
> 
> 									On Nov
> 27, 2015, at 7:23 PM, Josh Sokol <josh.sokol at owasp.org> wrote:
> 
> 
> 
> 
> 	Admittedly, this was my gut reaction at first as well.  I began linking all
> of these companies, people, and projects together in my mind (there are
> some loose links there) and painted a big conspiracy picture similar to what
> Jim and Dinis have stated.  But, after speaking directly with Jeff, and hearing
> about the conversation that Dave and Matt had, I've changed my mind.
> 
> 
> 
> 	I think it begins with the project itself.  If you aren't sold on the idea
> of the Benchmark, then you'll never be able to get to the same place.  My
> original line of thinking was that it was just a bar for vendors to compare their
> tools against eachother, but that's a bit myopic.  We are in an industry where
> things evolve very quickly.  As a customer of these tools, I know firsthand
> that something that a tool does today may not be the case a week from now.
> Likewise, new features are being added daily and I need a point-in-time
> metric to be able to gauge continual effectiveness.  Cool, right?  But not a
> game changer.  The game changer part comes when you realize that by
> developing and evolving the tests that go into the Benchmark, we are
> moving the bar higher and higher.  We (OWASP) are effectively setting the
> standard by which these tools will be compared.  A tool that receives a lower
> score on the Benchmark today knows exactly what they need to work on in
> order to pass that test tomorrow and we already have examples of tools that
> have made improvements because of their Benchmark score (Ask Simon
> about ZAP's experience with the Benchmark).  I don't think that anyone can
> argue that the Benchmark project isn't being effective when OWASP's own
> tools are being driven forward as a result of using it.
> 
> 
> 
> 	But, but, but, Dave and Jeff own Aspect and have stock in Contrast
> and Jeff is the Contrast CTO and Contrast got good scores so it's a conspiracy
> right?  Is there some code that allows Contrast to use the Benchmark?
> Absolutely.  Can you really blame Dave for starting his testing on the
> effectiveness of the Benchmark with a tool that he owned and is familiar
> with?  If I were going to start a similar project, there's no question in my mind
> that I would begin my testing with the tools that I have available to me.  That
> said, is there code that allows other tools to use the Benchmark?  Absolutely.
> 
> 
> 
> 	Regarding "Dave has a history of breaching his duty to be vendor
> neutral", while I cannot comment on his past actions, I can judge what we've
> seen recently.  Matt saw a presentation from Dave on the Benchmark at a
> conference in Chicago.  He said that he felt that the message was appropriate
> and while IAST tools were mentioned as receiving higher scores, it wasn't a
> "Contrast is the best" type of message, more of a generality.  I saw a very
> similar (if not the same) talk by Jeff at LASCON 2015 and the message was
> exactly the same.  I watched the talk expecting some sort of impropriety, but
> found none.  So, perhaps Dave has abused some privilege granted to him in
> the past, but what I've seen from him at this point, with respect to the
> Benchmark, has been appropriate.
> 
> 
> 
> 	You have a very good point with respect to the Contrast marketing
> message around the Benchmark.  It's been completely absurd, over the top,
> and, in my personal opinion, intolerable.  In fact, I experienced the same
> thing that you talked about with them at LASCON 2015 where they stood in
> front of the door of the room Jeff was speaking in and scanned attendees as
> they went into the talk.  I agree that these types of aggressive marketing
> tactics cannot be tolerated at OWASP.  In addition, we have seen several
> marketing messages from them effectively implying that OWASP endorses
> Contrast.  Clearly this is not OK.  I've spoken with Jeff about it and we agreed
> that it is not in the Benchmark's best interest to have this aggressive Contrast
> marketing around it at such an early stage.  He has said that he is not
> responsible for Contrast's marketing team, but that he would speak with the
> people who are.  I haven't seen a single message from them since so I'm
> guessing that he's made good on this promise.  While that's an excellent
> start, OWASP's takeaway here should be that we need to do a better job
> with our brand usage guidelines both in terms of the wording and
> enforcement.  There are many other companies out there that use the
> OWASP brand and I think that we agree that selective enforcement against
> Contrast is not the right answer.  Paul and Noreen are actively working on
> this.  Either way, I think that implying that activities from a vendor's
> marketing department means that the project is not objective is not
> inappropriate.  If we feel that the project is not objective, then separate
> measures need to be taken to drive contribution diversity into it.  That I
> absolutely agree with and the message from Dave was that he would love to
> have more contributors to his project.  But, seeing as we cannot force people
> to work on it, this becomes a matter of "put up or shut up".  The same goes
> for the experts that you said reviewed the code.  If they feel that it is
> somehow skewed towards Contrast, they have the power to change that.
> Now, if someone tries to participate and Dave tells them "No thanks", then I
> agree we have a problem, but I don't hear anyone inferring that happened.
> 
> 
> 
> 	Please, let's drop the conspiracy theories and focus on the tangible
> things that we can do to help an OWASP project to be more successful.  Help
> find more participants to drive diversity, update our brand usage guidelines
> to prevent abuse, enforce them widely, etc.  Thank you.
> 
> 
> 
> 	~josh
> 
> 
> 
> 	On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico
> <jim.manico at owasp.org> wrote:
> 
> 
> 
> 	Dinis,
> 
> 
> 	Like a rare celestial moment when all the planets plus Pluto are
> aligned, I just read your email on the future of OWASP projects thinking,
> "Dinis is spot on".
> 
> 
> 	Reflecting on projects I manage or work on...
> 
> 
> 	The Java Encoder and HTML Sanitizer are likely best moved to Apache
> now that they have reached a measure of adoption and maturity. Apache
> would be a much better long term custodian. Perhaps the same for
> AppSensor, but not my project - just thinking out loud.
> 
> 
> 	Other similar defensive projects are still being noodled on, so OWASP
> is a decent home for these research efforts.
> 
> 
> 	The whole tools category is also something to consider. Dependency
> Check and of course ZAP are some of the best projects that OWASP offers,
> are they best served where they are today? Both have rich communities of
> developers but I don't see the foundation doing much to support these
> efforts.
> 
> 
> 	ASVS has the opportunity to effect massive change, I would to love
> to see major investment and volunteer activity here. Pro tech writer,
> detailed discourses on each individual requirement, etc. If I was king (and I
> am not, at all) I would invest in ASVS on a 6 figure scale. (And who started
> ASVS? Jeff, Dave and Boberski, hat tip to such a marvelous idea). Or maybe
> moving ASVS to the W3C or IETF would help it grow?
> 
> 
> 	The Proactive Controls was a pet project but as we approach 2.0 we
> have several active/awesome volunteers working on it. We will be making
> the doc "world editable" to make contributions easy. OWASP seems like a
> good home for such an awareness doc. Same with T10, especially if
> community edits are welcome.
> 
> 
> 	Anyhow, I'm with you on this Dinis. Once a project starts to reach
> production quality, spinning off the project as an external project or moving
> it to a different foundation where managing production software or formal
> standards is their thing seems realistic.
> 
> 
> 	I don't have all the answers here, but your email certainly resonated
> with me.
> 
> 
> 	Aloha,
> 
> 	--
> 
> 	Jim Manico
> 
> 	Global Board Member
> 
> 	OWASP Foundation
> 
> 	https://www.owasp.org <https://www.owasp.org/>
> 
> 	Join me in Rome for AppSecEU 2016!
> 
> 
> 	On Nov 26, 2015, at 11:26 PM, Dinis Cruz <dinis.cruz at owasp.org>
> wrote:
> 
> 
> 
> 
> 
> 
> 		Jim's reading of this situation is exactly my view on the value
> of the Contrast tool and how it has been 'pushing' the rules of engagement
> to an very 'fuzzy' moral/ethical/commercial limit :)
> 
> 
> 		As per my last email, a key problem here is the 'perceived
> expectation' of what is an OWASP project, and how it should be consumed.
> 
> 
> 		If you look at the OWASP benchmark as a research project,
> then the only way it could be making the kind of claims it makes (and have
> credibility) is if it had evolved from OWASP, with its own (diverse) community
> 
> 
> 		On 26 November 2015 at 21:01, Jim Manico
> <jim.manico at owasp.org> wrote:
> 
> 
> 
> 
> 			I have a different take on this situation but my
> opinion is the "minority opinion". I will respect the rest of the boards take on
> this, but here is how I see it.
> 
> 
> 			First of all, Jeff has stated that he feels I am attacking
> him personally from a past personal grudge, and frankly I do not fault him for
> that perspective since we definitely have history with conflict. So it's fair to
> take my opinion on this with a grain of salt.
> 
> 
> 			I look at this situation from the perspective of a
> forensic investigator.
> 
> 
> 			1) The Benchmark project had Contrast hooks and
> only Contrast hooks in it when I reviewed it so this leads me to believe that
> the project was clearly built with Contrast in mind from the ground up, at
> least in some way.
> 
> 			3) Dave has a history of breaching his duty to be
> vendor neutral. He was gifted with a keynote in South Korea a few years ago,
> and used that opportunity to discuss and pitch Contrast, on stage, during a
> keynote - with Contrast specific slides. This is just supporting evidence of his
> intention at OWASP to push Contrast in ways that I think are against the
> intentions and goals of our foundation.
> 
> 			3) Other experts have reviewed the project and felt
> that many of the tests were very slanted and almost contrived to support
> Contrast. I can drag those folks into this conversation, but I do not think that
> would help in any way. So it's fair to call this point heresy.
> 
> 			4) I do not see this project as revolutionary, at all.
> Every vendor has their own test suite tuned for their tool. As the benchmark
> stands today, I see it as just another vendors product-specific benchmark.
> Mass collaboration from many vendors is not just a "nice to have" but a base
> requirement to get even close to useful for objective tool measurement.
> 
> 			5) Jeff stating that his Marketing people went over
> the line is also an admission that - well, they went over the line. By the same
> token Jeff was in his booth at AppSec USA surrounded by benchmark
> marketing material, discussing this to prospects and he even asked me and
> Mr Coates to wade into this debate and support Dave. So to say he was not
> involved and it was only his marketing people seems a stretch at best.
> 
> 			6) The Contrast marketing team was wandering
> around the conference zapping folks to get leads, and I asked them to stay in
> their booth, which is standard conference policy. These folks know better
> but are again going over the line to sell product at OWASP. There is a better
> way (like focusing on product capability and language support, have
> consistent + stellar customer service, have a humble and gracious attitude to
> all prospects and customers, actively participate in OWASP in a vendor
> neutral and community supportive way, etc).
> 
> 
> 			Please note, I think Contrast is a decent tool, I've
> offered to resell in the past, and I have recommended it in certain situations -
> even after this situation arose. I'm stating this out of honestly and desire to
> put my cards on the table. I truly want Jeff and Dave to be successful. They
> have dedicated their lives to AppSec and if anyone should win big-time, I
> hope it's them. I even told Jeff I hope he hits the mother load and donates a
> little back to OWASP.
> 
> 
> 			However, my instinct and evidence tell me that they
> both went over the line in the use of the OWASP brand to sell product.
> 
> 
> 			Now, Jeff makes a good point. We as a board and
> staff are very poor at enforcing brand management policy and it's not fair to
> single out Contrast, when many other vendors violate the brand, IMO. Just
> google OWASP and watch the ads fly that use the OWASP name to sell
> product.
> 
> 
> 			Also, any and every request that was made of Dave to
> adjust the project for the sake of vendor neutrality was taken very seriously.
> Regardless of Daves past intentions, he is clearly trying to do the right thing
> moving forward.
> 
> 
> 			I look to "postels principle" in this situation (this is
> otherwise known as the "robustness principle" and dates back to the
> creation of TCP) . This is paraphrased as, "Be liberal in what you take from
> others but be conservative in what you dish out". So I think it's critical that
> OWASP and any OWASP resource present itself in a strict vendor neutral
> way. But unless OWASP wants to be much more "even" in the enforcement
> of brand policy across the board to all violators, we should be fairly lax in the
> enforcement of these issues from the outside world.
> 
> 
> 			I am trying to be objective here. My trigonometry
> teacher once told me "I'd fail my mother" when I asked him if he would ever
> fail me (I was an A student). If my mother owned a security company and
> tried the same stunt, I'd have the same opinions about her actions as well.
> 
> 
> 			So what next? Well hello from the other side. I'm
> going back to listening to Adele's new album where I can sit in my deep
> feelings and reflect upon what the OWASP foundation has done to enrich my
> life. I would much rather keep out of this (and any other conflict laden
> situation at OWASP), but I feel it's my responsibility to speak up.
> 
> 
> 			Aloha,
> 
> 
> 
> 			--
> 
> 			Jim Manico
> 
> 			Global Board Member
> 
> 			OWASP Foundation
> 
> 			https://www.owasp.org <https://www.owasp.org/>
> 
> 			Join me in Rome for AppSecEU 2016!
> 
> 
> 			On Nov 26, 2015, at 9:09 PM, Josh Sokol
> <josh.sokol at owasp.org> wrote:
> 
> 
> 
> 
> 
> 
> 				I would be happy to provide an update.
> 
> 
> 
> 
> 				*	Matt Konda and Dave Wichers, the
> Benchmark Project Leader, had a conversation a few weeks back.  To
> summarize their conversation, Dave acknowledges the currently lack of
> diversity in his project and it is his sincere desire to drive more people to it to
> help.  He also acknowledges the issues with Contrast's extreme marketing
> around the project and feels that it is in everyone's best interests for them to
> curb it back.  While he does have an ownership stake in Contrast, he works at
> Aspect and has no control over the marketing messages that they are putting
> out there.  From the Board perspective, there has been no evidence of any
> impropriety on Dave's part and it should be our goal to drive more diversity
> into the project to support Dave.  Dave appears to be sincere in his desires to
> create a tool where OWASP can tell vendors what we expect from their
> tools.  If the main issue is that only members of Aspect are working on it,
> then the best thing that we can do is try to get him some outside assistance.
> We are also asking that the project be opened up to commits via Git so that
> outsiders can push commits to it.
> 
> 
> 
> 				*	Josh Sokol and Jeff Williams, the CTO
> of Contrast, had a conversation a few weeks back.  To summarize their
> conversation, Jeff believes that the work that Dave is doing on the
> Benchmark is a game changer in that it gives OWASP the power in dictating
> what these tools need to be finding.  He wants the Benchmark to be
> successful and understands that it needs to be diverse in order to be trusted.
> He recognizes that Dave is trying to do that and does not want the marketing
> message from Contrast to interfere with his efforts.  Jeff felt that the "Lab"
> status granted to Benchmark meant that it was ready for mainstream
> adoption, that it had 21k tests, and was almost a year old, and didn't see
> anything wrong with marketing their results, but has agreed to talk to their
> marketing team to get them to lay off that message for now.  From the Board
> perspective, we have come to the realization that our brand usage guidelines
> need an overhaul to clarify what is and is not allowed.  We have made a few
> proposals and have reached out to Mozilla to gain more insight on their
> guidelines and even ask for assistance.  Noreen and Paul are taking lead on
> these efforts.
> 
> 				*	There is a note in the notes that the
> Board was supposed to follow up with an open letter to the community and
> companies involved describing our review and actions.  I don't think that has
> happened so I will remind the person who took on that action item.
> 
> 
> 				I'm happy to answer any questions that you
> may have.
> 
> 
> 
> 
> 
> 				~josh
> 
> 
> 
> 
> 				On Thu, Nov 26, 2015 at 11:55 AM, Tobias
> <tobias.gondrom at owasp.org> wrote:
> 
> 
> 
> 
> 					There have been several
> conversations on that matter and a dedicated call. Unfortunately for personal
> reasons I could not attend the last call as it was at 04:00am my local time, but
> all other board members did participate.
> 
> 
> 
> 					Could please one of my fellow board
> members give an update.
> 
> 
> 
> 					Best, Tobias
> 
> 
> 
> 
> 
> 
> 
> 					On 26/11/15 18:04, Timo Goosen
> wrote:
> 
> 
> 
> 
> 						I would also like to know the
> answer to Simon's question. We need to get rid of bad apples in OWASP in
> my opinion, there are too many people just using the OWASP "name" or
> "brand" to improve their own financial situation or career.
> 
> 
> 						Regards.
> 
> 						Timo
> 
> 
> 						On Thu, Nov 26, 2015 at 1:13
> PM, psiinon <psiinon at gmail.com> wrote:
> 
> 
> 
> 
> 							Paul, and the rest of
> the board,
> 
> 
> 
> 
> 
> 							Its been over 2 months
> since I raised this issue.
> 
> 
> 
> 							Whats happening?
> 
> 
> 
> 							Has the board even
> discussed it?
> 
> 
> 
> 
> 
> 							Cheers,
> 
> 
> 
> 
> 
> 							Simon
> 
> 
> 
> 
> 
> 							On Tue, Oct 20, 2015 at
> 10:00 PM, Paul Ritchie <paul.ritchie at owasp.org> wrote:
> 
> 
> 
> 
> 								Eoin, Johanna,
> All:
> 
> 
> 								In an earlier
> email, Josh Sokol mentioned that he will be speaking in the next day or 2 to
> their CTO, while at LASCON, as a representative of the OWASP Board.
> Following that feedback, the Board has action to take the next steps.
> 
> 
> 								Just an FYI that
> all comments are recognized and action is being taken.
> 
> 
> 								Paul
> 
> 
> 
> 
> 								Best Regards,
> Paul Ritchie
> 
> 								OWASP
> Executive Director
> 
> 
> 	paul.ritchie at owasp.org
> 
> 
> 
> 								On Tue, Oct 20,
> 2015 at 1:54 PM, johanna curiel curiel <johanna.curiel at owasp.org> wrote:
> 
> 
> 
> 
> 									Time
> for owasp to do a public statement and put a clear story regarding this
> abusive behavior of Owasp brand
> 
> 
> 
> 									On
> Tuesday, October 20, 2015, Eoin Keary <eoin.keary at owasp.org> wrote:
> 
> 
> 
> 
> 
> 	Folks,
> 
> 
> 
> 	The project should be immediately shelved it's simply bad form.
> 
> 
> 
> 	This is damaging to OWASP, the industry and exactly what OWASP is
> not about.
> 
> 
> 
> 	There is a clear conflict of interest and distinct lack of science behind
> the claims made by Contrast.
> 
> 
> 
> 
> 
> 
> 
> 
> 	Eoin Keary
> 
> 
> 	OWASP Volunteer
> 
> 
> 	@eoinkeary
> 
> 
> 
> 
> 
> 
> 
> 
> 	On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel
> <johanna.curiel at owasp.org> wrote:
> 
> 
> 
> 
> 
> 
> 
> 	At the moment we did the project review, we observed that the
> project did not have enough testing to be considered in any form as 'ready'
> for benchmarking, neither that it had yet the community adoption, however
> technically speaking as it has been classified by the leaders, the project is at
> the beta stage.
> 
> 
> 
> 	Indeed , Dave had the push to have the project reviewed but it was
> never clear that later on the project was going to be advertisied this way.
> That all happend after the presentation at Appsec.
> 
> 
> 
> 	I had my concerns regarding how sensitive is the subject of the
> project ,but I think we should allow project leaders to develop their
> communication strategy even if this has conflict of interest. It all depends
> how they behave and how they manage this.
> 
> 
> 
> 
> 	On Tuesday, October 6, 2015, Michael Coates
> <michael.coates at owasp.org> wrote:
> 
> 
> 
> 
> 
> 
> 		It's not really that formal to add to the agenda, just a wiki that
> we add in the text.
> 
> 
> 
> 		I think you can safely assume it will get the appropriate
> discussion.
> 
> 
> 
> 		On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com>
> wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 			Really?? Its not on the agenda yet for the next
> meeting??
> 
> 
> 			How does it get added to the agenda?
> 
> 
> 
> 
> 
> 			And that was a formal request if that makes any
> difference :)
> 
> 
> 
> 
> 
> 			I'm all in favour of getting the facts straight before any
> actions are taken, hence my request for an 'ethical review' or whatever it
> should be called.
> 
> 
> 
> 
> 
> 
> 
> 
> 			Cheers,
> 
> 
> 
> 
> 
> 
> 
> 
> 			Simon
> 
> 
> 
> 
> 
> 
> 			On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates
> <michael.coates at owasp.org> wrote:
> 
> 
> 
> 
> 
> 
> 				First step is to get all of our information
> straight so we're clear on where things are at.
> 
> 
> 
> 				This was not on the board agenda last
> meeting and is also not on the next agenda as of yet (of course it could
> always be added if needed).
> 
> 
> 
> 				We are aware that people have raised
> questions though.   I'm hoping we can get a clear understanding of all the
> facts and then discuss if changes are needed.
> 
> 
> 
> 
> 
> 				On Oct 6, 2015, at 1:52 AM, psiinon
> <psiinon at gmail.com> wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 					Hey Michael,
> 
> 
> 
> 
> 
> 					Is the board going to take any action?
> 
> 
> 
> 
> 
> 					Were there any discussions about this
> controversy in the board meeting at AppSec USA?
> 
> 
> 
> 
> 
> 					If not will it be on the agenda for the
> meeting on October 14th?
> 
> 
> 
> 
> 
> 
> 
> 
> 					Cheers,
> 
> 
> 
> 
> 
> 
> 
> 
> 					Simon
> 
> 
> 
> 
> 
> 
> 
> 					On Tue, Oct 6, 2015 at 8:25 AM,
> Michael Coates <michael.coates at owasp.org> wrote:
> 
> 
> 
> 
> 
> 
> 						Simon
> 
> 
> 
> 						I posted the below message
> earlier today. At this point my goal is to just gain clarity over the current
> reality and ideally drive to a shared state of success. This message doesn't
> seem to be reflected in the list yet. It could be because my membership
> hasn't been approved or because of mail list delays (I miss Google groups).
> But I think these questions will start the conversation.
> 
> 
> 
> 						(This was just me asking
> questions as a curious Owasp member, not any action on behalf of the
> board)
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 						Begin forwarded message:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 							From: Michael Coates
> <michael.coates at owasp.org>
> 
> 
> 							Date: October 5, 2015
> at 6:20:23 PM PDT
> 
> 
> 							To: owasp-benchmark-
> project at lists.owasp.org
> 
> 
> 							Subject: Project
> Questions
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 							OWASP Benchmark
> List,
> 
> 
> 
> 
> 
> 
> 
> 
> 							I've heard more about
> this project and am excited about the idea of an independent perspective of
> tool performance. I'm trying to understand a few things to better respond to
> questions from those in the security & OWASP community.
> 
> 
> 
> 
> 
> 
> 
> 
> 							In my mind there are
> two big areas for consideration in a benchmark process.
> 
> 
> 							1. Are the benchmarks
> testing the right areas?
> 
> 
> 							2. Is the process for
> creating the benchmark objective & free from conflicts of interest.
> 
> 
> 
> 
> 
> 
> 
> 
> 							I think as a group
> OWASP is the right body to align on #1.
> 
> 
> 
> 
> 
> 
> 
> 
> 							I'd like to ask for some
> clarifications on item #2. I think it's important to avoid actual conflict of
> interest and also the appearance of conflict of interest. The former is obvious
> why we mustn't have that, the latter is critical so others have faith in the tool,
> process and outputs of the process when viewing or hearing about the
> project.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 							1) Can we clarify
> whether other individuals have submitted meaningful code to the project?
> 
> 
> 							Observation:
> 
> 
> 							Nearly all the code
> commits have come from 1 person (project lead).
> 
> 
> 
> 	https://github.com/OWASP/Benchmark/graphs/contributors
> <https://github.com/OWASP/Benchmark/graphs/contributors>
> 
> 
> 
> 
> 
> 
> 
> 
> 							2) Can we clarify the
> contributions of others and their represented organizations?
> 
> 
> 							Observation:
> 
> 
> 							The
> acknowledgements tab listed two developers (Juan Gama & Nick Sanidas)
> both who work at the same company as the project lead. It seems other
> people have submitted some small amounts of material, but overall it seems
> all development has come from the same company.
> 
> 
> 
> 	https://www.owasp.org/index.php/Benchmark#tab=Acknowledgem
> ents
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 							3) Can we clarify in
> what ways we've mitigated the potential conflict of interest and also the
> appearance of a conflict of interest? This seems like the largest blocker for
> wide spread acceptance of this project and the biggest risk.
> 
> 
> 							Observation:
> 
> 
> 							The project lead and
> both of the project developers works for a company with very close ties to
> one of the companies that is evaluated by this project. Further, it appears
> the company is performing very well on the project tests.
> 
> 
> 
> 
> 
> 
> 
> 
> 							4) If we are going to list
> tool vendors then I'd recommend listing multiple vendors for each category.
> 
> 
> 							Observation:
> 
> 
> 							The tools page only
> lists 1 IAST tool. Since this is the point of the potential conflict of interest it is
> important to list numerous IAST tools.
> 
> 
> 
> 	https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_
> 2FResults
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 							5) Diverse body with
> multiple points of view
> 
> 
> 							Observation:
> 
> 
> 							There is no indication
> that multiple stakeholders are present to review and decide on the future of
> this project. If they exist, a new section should be added to the project page
> to raise awareness. If they don't exist, we should reevaluate how we are
> obtaining an independent view of the testing process.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 							Again, I think the idea
> of the project is great. From my perspective clarifying these questions will
> help ensure the project is not only objective, but also perceived as objective
> from someone reviewing the material. Ultimately this will contribute to the
> success and growth of the project.
> 
> 
> 
> 
> 
> 
> 
> 
> 							Thanks!
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 							--
> 
> 
> 							Michael Coates
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 						On Oct 2, 2015, at 1:31 AM,
> psiinon <psiinon at gmail.com> wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 							OK, based on the
> concerns raised so far I think the board should initiate a review of the OWASP
> Benchmark project.
> 
> 
> 
> 
> 
> 							I'm not raising a formal
> complaint against it, I'm just requesting a review.
> 
> 
> 
> 
> 
> 							And I dont think it
> needs a 'standard' project review - Johanna has already done a very good job
> of this.
> 
> 
> 
> 
> 
> 							Not sure what sort of
> review you'd call it, I'll leave the naming to others :)
> 
> 
> 
> 
> 
> 
> 
> 
> 							I'm concerned that we
> have an OWASP project lead by a company who has a clear commercial stake
> in the results.
> 
> 
> 
> 
> 
> 							Bringing more
> companies on board will help, but I'm still not sure that alone will make it
> independent enough.
> 
> 
> 
> 
> 
> 							Commercial companies
> can afford to dedicate staff to improving Benchmark so that their products
> look better.
> 
> 
> 
> 
> 
> 							Open source projects
> just cant do that, so we are at a distinct disadvantage.
> 
> 
> 
> 
> 
> 							Should we allow a
> commercially driven OWASP project who's aim could be seen be to promote
> commercial software?
> 
> 
> 
> 
> 
> 							If so, what sort of
> checks and balances does it need?
> 
> 
> 
> 
> 
> 							Those are the sort of
> questions I'd like an independent review to look at.
> 
> 
> 
> 
> 
> 
> 							I do think there are
> some immediate steps that could be taken:
> 
> 
> 
> 
> 
> 
> 							*	I'd like to see
> the Benchmark project page clearly state thats its at a very early stage and
> that the results are _not_ yet suitable for use in commercial literature.
> 
> 
> 							*	I'd also like the
> main companies developing Benchmark to be clearly stated on the main
> page. If and when other companies get involved then this would actually
> help the project's claim of vendor independence.
> 
> 
> 							*	And I'd love to
> see a respected co-leader added to the project who is not associated with
> any commercial or open source security tools:)
> 
> 
> 
> 							And we should carry
> on discussing the project on this list - I think such discussions are very
> healthy, and I'd love to see this project mature to a state where it can be a
> trusted, independent and valued resource.
> 
> 
> 
> 
> 
> 
> 							Cheers,
> 
> 
> 
> 
> 
> 
> 
> 
> 							Simon
> 
> 
> 
> 
> 
> 
> 							On Thu, Oct 1, 2015 at
> 7:59 PM, Tobias <tobias.gondrom at owasp.org> wrote:
> 
> 
> 
> 
> 
> 
> 								@Simon:
> 
> 
> 								yes, the
> leaders list is the place for your discussions for project and chapter leaders
> 
> 
> 								@Timo: I like
> your framing of "Don't ask what OWASP can do for me, ask what I can do for
> OWASP."
> 
> 
> 								That should
> and is indeed the spirit of OWASP:-)
> 
> 
> 								Best regards,
> Tobias
> 
> 
> 
> 
> 
> 
> 								On 30/09/15
> 09:42, Timo Goosen wrote:
> 
> 
> 
> 
> 	...
> 
> 	[Message clipped]
> 	_______________________________________________
> 	Owasp-board mailing list
> 	Owasp-board at lists.owasp.org
> 	https://lists.owasp.org/mailman/listinfo/owasp-board
> 
> 
> 
> 
> 
> 
> --
> 
> OWASP ZAP <https://www.owasp.org/index.php/ZAP>  Project leader