[Owasp-leaders] [Owasp-board] OWASP Benchmark project - potential conflict of interest

Claudia Casanovas claudia.aviles-casanovas at owasp.org
Mon Nov 30 17:32:40 UTC 2015


Hi All,

I can add the status as In Dispute Banner to the Benchmark Wiki Page
immediately.

Claudia

On Mon, Nov 30, 2015 at 9:27 AM, Tobias Glemser <tobias.glemser at owasp.org>
wrote:

> > At the _very_ least it should flag the project as being 'in dispute' (as
> Kevin
> > suggested) while a more detailed evaluation is performed.
> +1
>
> The conflict is clear scrolling through all those E-Mails, Blogs, etc.
> Until it's sorted out we need a clarification visible to everyone, the
> Benchmark projects status is heavily discussed within the Community at the
> Moment.
>
> > -----Ursprüngliche Nachricht-----
> > Von: owasp-leaders-bounces at lists.owasp.org [mailto:owasp-leaders-
> > bounces at lists.owasp.org] Im Auftrag von psiinon
> > Gesendet: Montag, 30. November 2015 18:18
> > An: Jim Manico
> > Cc: OWASP Foundation Board List; owasp-leaders at lists.owasp.org
> > Betreff: Re: [Owasp-leaders] [Owasp-board] OWASP Benchmark project -
> > potential conflict of interest [ Z1 UNGESICHERT ]
> >
> > I'd like to start by saying that I actually _like_ the Benchmark project.
> > Myself and other ZAP developers have made some contributions to it, and
> > we have used (and will continue to use) it to make ZAP better.
> > I think these sort of testing applications are very valuable to all
> security tools,
> > and I'd like to thank Dave and his team for the significant amount of
> effort
> > involved in developing and open sourcing it.
> >
> > But I dont think it should be an OWASP project.
> > I do not think that a vendor led project can ever objectively evaluate
> > competing commercial and open source projects.
> > I do not think that just saying 'pull requests welcomed' makes a project
> > vendor neutral.
> > I do not think that a project as mired in controversy as the Benchmark
> project
> > can ever recover to become truly independent.
> >
> > I am very disappointed in the Boards handling of this affair.
> >
> > Ideally I'd like Dave to understand how much damage this project has done
> > and to withdraw it as an OWASP project, while still maintaining it as a
> very
> > valuable vendor led open source resource.
> >
> > Failing that I really hope that the Board comes to its senses and ejects
> the
> > Benchmark project before even more damage is done.
> > At the _very_ least it should flag the project as being 'in dispute' (as
> Kevin
> > suggested) while a more detailed evaluation is performed.
> >
> > However I'm rapidly loosing loosing faith that the Board will do the
> right thing
> > and protect OWASP's image in the way that they should have already done.
> > Members - please make your voices heard before more people and projects
> > leave OWASP.
> >
> > Simon
> >
> >
> > On Sat, Nov 28, 2015 at 5:14 AM, Jim Manico <jim.manico at owasp.org>
> > wrote:
> >
> >
> >       WAFEC does not "do vendor assessment"; they define a
> > comprehensive standard built by many vendors and let the community use
> > that standard to measure tools on their own. Just a FYI, I was involved
> in the
> > early version of this project. (Things may have changed since my
> > involvement, I'm sure Tony has more details here)
> >
> >       Johanna's comments on this issue lead me to believe that the
> > damage done to both OWASP and DHS is even more destructive that I
> > thought. It saddens me to see this level of abuse just to sell product.
> >
> >
> >       --
> >       Jim Manico
> >       Global Board Member
> >       OWASP Foundation
> >       https://www.owasp.org <https://www.owasp.org/>
> >       Join me in Rome for AppSecEU 2016!
> >
> >       On Nov 28, 2015, at 2:40 AM, Josh Sokol <josh.sokol at owasp.org>
> > wrote:
> >
> >
> >
> >               One of the ideas that Andrew proposed was actually
> > approaching WAFEC to learn more about how they do vendor assessment in
> > a neutral way.  It's great to hear that we have a resource here already
> that
> > we can leverage.  I wasn't aware of your affiliation.
> >
> >               ~josh
> >
> >               On Nov 27, 2015 2:47 PM, "Tony Turner"
> > <tony.turner at owasp.org> wrote:
> >
> >
> >                       I sincerely hope so. That's not the impression I
> got
> > from others comments. Personally I haven't used the tool at all, but as
> I'm
> > the project lead for another product evaluation project (WAFEC) I'm very
> > sensitive to the need of collaboration with many different vendors. There
> > really has to be a very high level (almost paranoid level) transparency
> with
> > how vendors are approached, worked with, how requirements for
> > evaluation are defined, and how metrics are derived.
> >
> >                       It appears the project team is attempting to
> address
> > these last 2 somewhat but I'd like to see more specifics, and the lack of
> > information on how they are addressing vendor communication,
> > participation and transparency seems a bit concerning. Lastly, it is my
> opinion
> > that project leadership should not belong to anyone working for or with a
> > partnership/ownership stake for any vendor being evaluated. I think this
> is a
> > flawed model and should transition to a vendor neutral party.
> >
> >                       On Nov 27, 2015 3:16 PM, "Josh Sokol"
> > <josh.sokol at owasp.org> wrote:
> >
> >
> >                               I don't know what qualifies as
> "significant" in
> > your mind, but my understanding is that there have been contributions
> from
> > other vendors:
> >
> >
> >       https://www.owasp.org/index.php/Benchmark#tab=Acknowledgem
> > ents
> >
> >
> >                               Still, Dave would like more, but he can't
> force
> > them to help.
> >
> >
> >                               ~josh
> >
> >
> >                               On Fri, Nov 27, 2015 at 1:45 PM, Tony
> Turner
> > <tony.turner at owasp.org> wrote:
> >
> >
> >                                       While I can appreciate that they
> > started with Contrast, if there hasn't been significant effort to
> include other
> > vendors it's a worthless benchmark. It's easy to state you haven't gotten
> > support from other vendors and that's fine, but until you do there's
> really
> > nothing to release. Why was it ever upgraded? Talking about the results
> > without an accurate comparative analysis is akin to snake oil.
> >
> >                                       On Nov 27, 2015 1:49 PM, "Josh
> Sokol"
> > <josh.sokol at owasp.org> wrote:
> >
> >
> >                                               Thank you for the links to
> > those articles.  The first one discusses the strengths and weaknesses of
> the
> > different methods of evaluating for application vulnerabilities.  The
> section
> > on the Benchmark seems wholly appropriate to me.  That seems like an
> > excellent description of what the project is designed to do.  I see some
> > metrics in there about which tools are more effective on which types of
> > vulnerabilities, but I don't see him straight up saying "The OWASP
> Benchmark
> > proves that Contrast is better".  This seems like statements made based
> on
> > some level of testing and research.  Honestly, I don't see any OWASP
> brand
> > abuse in that article.  Whether it's in good taste or not at this stage
> in the
> > project is certainly debatable, but if you look at the brand usage
> guidelines
> > (https://www.owasp.org/index.php/Marketing/Resources#tab=BRAND_GU
> > IDELINES), I don't see any violations.  We need to govern to policy here
> which
> > is why Paul and Noreen are evaluating changes to the guidelines and our
> > enforcement policies to make abuse more difficult.
> >
> >
> >                                               The second article is a
> > competing vendor's reaction to the first.  He makes some good points
> about
> > the issues with Benchmark, but he also says that he hopes that it will be
> > improved over time, and Dave has committed to that.  What I don't see is
> the
> > vendor saying "...and Veracode has committed resources to help make the
> > Benchmark more accurate across all tool sets".  The Benchmark page is
> pretty
> > clear that it does it's best to provide a benchmark without working
> exactly
> > like a real-world application.  Maybe some more disclaimer text about
> where
> > the project is at today would be in order to validate some of Chris'
> concerns,
> > but I hardly see this as "brand abuse" or a reason to demote the project.
> >
> >
> >                                               Please consider that I have
> > spoken with both Dave and Jeff on this topic and read much of the
> > discussions around it before formulating my opinion.  I doubt that you
> have
> > done the same so I'm not sure how you can claim that you have researched
> > the issues and all parties involved when you haven't even spoken with the
> > two people whom you are accusing of impropriety.  I have no bias here.
> I am
> > simply speaking with the individuals involved, looking at the currently
> OWASP
> > policies and guidelines, and helping to determine our next steps.
> >
> >
> >                                               ~josh
> >
> >
> >                                               On Fri, Nov 27, 2015 at
> 12:22
> > PM, johanna curiel curiel <johanna.curiel at owasp.org> wrote:
> >
> >
> >                                                       >>While I agree
> with
> > you that there has been some brand abuse, it was abuse by Contrast
> > (specifically their marketing department), and not by "these gentlemen"
> as
> > you state.
> >
> >                                                       Really? ..'some
> brand
> > abuse'..this is more than brand abuse
> >
> >
> >
> >                                                       Josh , please read
> also
> > the article written by Jeff
> >
> >       http://www.darkreading.com/vulnerabilities---threats/why-its-
> > insane-to-trust-static-analysis/a/d-id/1322274?
> >
> >
> >
> >                                                       And Veracode's
> > reaction including others in Twitter
> >
> >       https://www.veracode.com/blog/2015/09/no-one-technology-
> > silver-bullet
> >
> >
> >                                                       My strong advice
> is to
> > research the issues and all the parties involved before making statements
> >
> >
> >
> >
> >                                                       On Fri, Nov 27,
> 2015 at
> > 2:07 PM, Josh Sokol <josh.sokol at owasp.org> wrote:
> >
> >
> >                                                               Jim,
> >
> >
> >                                                               A concern
> was
> > expressed to the Board and, frankly, I am insulted by you saying that
> this was
> > "brushed under the rug".  The Board delegated Matt to talk with Dave and
> > they had a lengthy conversation on the subject.  The Board delegated me
> to
> > talk with Jeff and we had a lengthy conversation on the subject.  If you
> do
> > not trust in our abilities to read people, ask the right questions, and
> provide
> > honest feedback about our conversations, then that's a bigger issue that
> we
> > should take offline.  After our conversations, we took the time to call a
> > special two-hour session of the Board in order to discuss this subject
> (and
> > only this subject).  We spoke about all facets of the issue at hand,
> about the
> > challenges and possible solutions, and concluded on some very concrete
> > next steps.
> >
> >                                                               While I
> agree
> > with you that there has been some brand abuse, it was abuse by Contrast
> > (specifically their marketing department), and not by "these gentlemen"
> as
> > you state.  Unless you can point to some sort of evidence showing that
> Jeff
> > and/or Dave first-hand abused the brand, then I believe that you are
> > speaking with your heart instead of with your head.  I appreciate your
> > passion, but I label this as conspiracy theory because without evidence
> to
> > support your claims, I cannot accept it as anything other.
> >
> >
> >                                                               ~josh
> >
> >
> >                                                               On Fri,
> Nov 27,
> > 2015 at 11:39 AM, Jim Manico <jim.manico at owasp.org> wrote:
> >
> >
> >
>  Josh,
> >
> >                                                                       I
> stand
> > by my comments and perspective, but I'm disheartened that you consider
> > my presentation of facts (and the concerns of many active members of our
> > community) as a "conspiracy theory".
> >
> >                                                                       In
> my
> > experience, these kind of comments border on insults and only cause folks
> > to harden their opinions.
> >
> >
>  Once
> > again I feel these gentlemen got away with a kind of brand abuse that is
> very
> > hurtful to the OWASP community but I am at a loss as to how handle or
> > prevent these kinds of mishaps - especially when board members like
> > yourself seem willing to - from what I see - brush it under the rug.
> >
> >
> >                                                                       --
> >                                                                       Jim
> > Manico
> >
>  Global
> > Board Member
> >
>  OWASP
> > Foundation
> >
> >       https://www.owasp.org <https://www.owasp.org/>
> >
>  Join me
> > in Rome for AppSecEU 2016!
> >
> >                                                                       On
> Nov
> > 27, 2015, at 7:23 PM, Josh Sokol <josh.sokol at owasp.org> wrote:
> >
> >
> >
> >
> >       Admittedly, this was my gut reaction at first as well.  I began
> linking all
> > of these companies, people, and projects together in my mind (there are
> > some loose links there) and painted a big conspiracy picture similar to
> what
> > Jim and Dinis have stated.  But, after speaking directly with Jeff, and
> hearing
> > about the conversation that Dave and Matt had, I've changed my mind.
> >
> >
> >
> >       I think it begins with the project itself.  If you aren't sold on
> the idea
> > of the Benchmark, then you'll never be able to get to the same place.  My
> > original line of thinking was that it was just a bar for vendors to
> compare their
> > tools against eachother, but that's a bit myopic.  We are in an industry
> where
> > things evolve very quickly.  As a customer of these tools, I know
> firsthand
> > that something that a tool does today may not be the case a week from
> now.
> > Likewise, new features are being added daily and I need a point-in-time
> > metric to be able to gauge continual effectiveness.  Cool, right?  But
> not a
> > game changer.  The game changer part comes when you realize that by
> > developing and evolving the tests that go into the Benchmark, we are
> > moving the bar higher and higher.  We (OWASP) are effectively setting the
> > standard by which these tools will be compared.  A tool that receives a
> lower
> > score on the Benchmark today knows exactly what they need to work on in
> > order to pass that test tomorrow and we already have examples of tools
> that
> > have made improvements because of their Benchmark score (Ask Simon
> > about ZAP's experience with the Benchmark).  I don't think that anyone
> can
> > argue that the Benchmark project isn't being effective when OWASP's own
> > tools are being driven forward as a result of using it.
> >
> >
> >
> >       But, but, but, Dave and Jeff own Aspect and have stock in Contrast
> > and Jeff is the Contrast CTO and Contrast got good scores so it's a
> conspiracy
> > right?  Is there some code that allows Contrast to use the Benchmark?
> > Absolutely.  Can you really blame Dave for starting his testing on the
> > effectiveness of the Benchmark with a tool that he owned and is familiar
> > with?  If I were going to start a similar project, there's no question
> in my mind
> > that I would begin my testing with the tools that I have available to
> me.  That
> > said, is there code that allows other tools to use the Benchmark?
> Absolutely.
> >
> >
> >
> >       Regarding "Dave has a history of breaching his duty to be vendor
> > neutral", while I cannot comment on his past actions, I can judge what
> we've
> > seen recently.  Matt saw a presentation from Dave on the Benchmark at a
> > conference in Chicago.  He said that he felt that the message was
> appropriate
> > and while IAST tools were mentioned as receiving higher scores, it
> wasn't a
> > "Contrast is the best" type of message, more of a generality.  I saw a
> very
> > similar (if not the same) talk by Jeff at LASCON 2015 and the message was
> > exactly the same.  I watched the talk expecting some sort of
> impropriety, but
> > found none.  So, perhaps Dave has abused some privilege granted to him in
> > the past, but what I've seen from him at this point, with respect to the
> > Benchmark, has been appropriate.
> >
> >
> >
> >       You have a very good point with respect to the Contrast marketing
> > message around the Benchmark.  It's been completely absurd, over the top,
> > and, in my personal opinion, intolerable.  In fact, I experienced the
> same
> > thing that you talked about with them at LASCON 2015 where they stood in
> > front of the door of the room Jeff was speaking in and scanned attendees
> as
> > they went into the talk.  I agree that these types of aggressive
> marketing
> > tactics cannot be tolerated at OWASP.  In addition, we have seen several
> > marketing messages from them effectively implying that OWASP endorses
> > Contrast.  Clearly this is not OK.  I've spoken with Jeff about it and
> we agreed
> > that it is not in the Benchmark's best interest to have this aggressive
> Contrast
> > marketing around it at such an early stage.  He has said that he is not
> > responsible for Contrast's marketing team, but that he would speak with
> the
> > people who are.  I haven't seen a single message from them since so I'm
> > guessing that he's made good on this promise.  While that's an excellent
> > start, OWASP's takeaway here should be that we need to do a better job
> > with our brand usage guidelines both in terms of the wording and
> > enforcement.  There are many other companies out there that use the
> > OWASP brand and I think that we agree that selective enforcement against
> > Contrast is not the right answer.  Paul and Noreen are actively working
> on
> > this.  Either way, I think that implying that activities from a vendor's
> > marketing department means that the project is not objective is not
> > inappropriate.  If we feel that the project is not objective, then
> separate
> > measures need to be taken to drive contribution diversity into it.  That
> I
> > absolutely agree with and the message from Dave was that he would love to
> > have more contributors to his project.  But, seeing as we cannot force
> people
> > to work on it, this becomes a matter of "put up or shut up".  The same
> goes
> > for the experts that you said reviewed the code.  If they feel that it is
> > somehow skewed towards Contrast, they have the power to change that.
> > Now, if someone tries to participate and Dave tells them "No thanks",
> then I
> > agree we have a problem, but I don't hear anyone inferring that happened.
> >
> >
> >
> >       Please, let's drop the conspiracy theories and focus on the
> tangible
> > things that we can do to help an OWASP project to be more successful.
> Help
> > find more participants to drive diversity, update our brand usage
> guidelines
> > to prevent abuse, enforce them widely, etc.  Thank you.
> >
> >
> >
> >       ~josh
> >
> >
> >
> >       On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico
> > <jim.manico at owasp.org> wrote:
> >
> >
> >
> >       Dinis,
> >
> >
> >       Like a rare celestial moment when all the planets plus Pluto are
> > aligned, I just read your email on the future of OWASP projects thinking,
> > "Dinis is spot on".
> >
> >
> >       Reflecting on projects I manage or work on...
> >
> >
> >       The Java Encoder and HTML Sanitizer are likely best moved to Apache
> > now that they have reached a measure of adoption and maturity. Apache
> > would be a much better long term custodian. Perhaps the same for
> > AppSensor, but not my project - just thinking out loud.
> >
> >
> >       Other similar defensive projects are still being noodled on, so
> OWASP
> > is a decent home for these research efforts.
> >
> >
> >       The whole tools category is also something to consider. Dependency
> > Check and of course ZAP are some of the best projects that OWASP offers,
> > are they best served where they are today? Both have rich communities of
> > developers but I don't see the foundation doing much to support these
> > efforts.
> >
> >
> >       ASVS has the opportunity to effect massive change, I would to love
> > to see major investment and volunteer activity here. Pro tech writer,
> > detailed discourses on each individual requirement, etc. If I was king
> (and I
> > am not, at all) I would invest in ASVS on a 6 figure scale. (And who
> started
> > ASVS? Jeff, Dave and Boberski, hat tip to such a marvelous idea). Or
> maybe
> > moving ASVS to the W3C or IETF would help it grow?
> >
> >
> >       The Proactive Controls was a pet project but as we approach 2.0 we
> > have several active/awesome volunteers working on it. We will be making
> > the doc "world editable" to make contributions easy. OWASP seems like a
> > good home for such an awareness doc. Same with T10, especially if
> > community edits are welcome.
> >
> >
> >       Anyhow, I'm with you on this Dinis. Once a project starts to reach
> > production quality, spinning off the project as an external project or
> moving
> > it to a different foundation where managing production software or formal
> > standards is their thing seems realistic.
> >
> >
> >       I don't have all the answers here, but your email certainly
> resonated
> > with me.
> >
> >
> >       Aloha,
> >
> >       --
> >
> >       Jim Manico
> >
> >       Global Board Member
> >
> >       OWASP Foundation
> >
> >       https://www.owasp.org <https://www.owasp.org/>
> >
> >       Join me in Rome for AppSecEU 2016!
> >
> >
> >       On Nov 26, 2015, at 11:26 PM, Dinis Cruz <dinis.cruz at owasp.org>
> > wrote:
> >
> >
> >
> >
> >
> >
> >               Jim's reading of this situation is exactly my view on the
> value
> > of the Contrast tool and how it has been 'pushing' the rules of
> engagement
> > to an very 'fuzzy' moral/ethical/commercial limit :)
> >
> >
> >               As per my last email, a key problem here is the 'perceived
> > expectation' of what is an OWASP project, and how it should be consumed.
> >
> >
> >               If you look at the OWASP benchmark as a research project,
> > then the only way it could be making the kind of claims it makes (and
> have
> > credibility) is if it had evolved from OWASP, with its own (diverse)
> community
> >
> >
> >               On 26 November 2015 at 21:01, Jim Manico
> > <jim.manico at owasp.org> wrote:
> >
> >
> >
> >
> >                       I have a different take on this situation but my
> > opinion is the "minority opinion". I will respect the rest of the boards
> take on
> > this, but here is how I see it.
> >
> >
> >                       First of all, Jeff has stated that he feels I am
> attacking
> > him personally from a past personal grudge, and frankly I do not fault
> him for
> > that perspective since we definitely have history with conflict. So it's
> fair to
> > take my opinion on this with a grain of salt.
> >
> >
> >                       I look at this situation from the perspective of a
> > forensic investigator.
> >
> >
> >                       1) The Benchmark project had Contrast hooks and
> > only Contrast hooks in it when I reviewed it so this leads me to believe
> that
> > the project was clearly built with Contrast in mind from the ground up,
> at
> > least in some way.
> >
> >                       3) Dave has a history of breaching his duty to be
> > vendor neutral. He was gifted with a keynote in South Korea a few years
> ago,
> > and used that opportunity to discuss and pitch Contrast, on stage,
> during a
> > keynote - with Contrast specific slides. This is just supporting
> evidence of his
> > intention at OWASP to push Contrast in ways that I think are against the
> > intentions and goals of our foundation.
> >
> >                       3) Other experts have reviewed the project and felt
> > that many of the tests were very slanted and almost contrived to support
> > Contrast. I can drag those folks into this conversation, but I do not
> think that
> > would help in any way. So it's fair to call this point heresy.
> >
> >                       4) I do not see this project as revolutionary, at
> all.
> > Every vendor has their own test suite tuned for their tool. As the
> benchmark
> > stands today, I see it as just another vendors product-specific
> benchmark.
> > Mass collaboration from many vendors is not just a "nice to have" but a
> base
> > requirement to get even close to useful for objective tool measurement.
> >
> >                       5) Jeff stating that his Marketing people went over
> > the line is also an admission that - well, they went over the line. By
> the same
> > token Jeff was in his booth at AppSec USA surrounded by benchmark
> > marketing material, discussing this to prospects and he even asked me and
> > Mr Coates to wade into this debate and support Dave. So to say he was not
> > involved and it was only his marketing people seems a stretch at best.
> >
> >                       6) The Contrast marketing team was wandering
> > around the conference zapping folks to get leads, and I asked them to
> stay in
> > their booth, which is standard conference policy. These folks know better
> > but are again going over the line to sell product at OWASP. There is a
> better
> > way (like focusing on product capability and language support, have
> > consistent + stellar customer service, have a humble and gracious
> attitude to
> > all prospects and customers, actively participate in OWASP in a vendor
> > neutral and community supportive way, etc).
> >
> >
> >                       Please note, I think Contrast is a decent tool,
> I've
> > offered to resell in the past, and I have recommended it in certain
> situations -
> > even after this situation arose. I'm stating this out of honestly and
> desire to
> > put my cards on the table. I truly want Jeff and Dave to be successful.
> They
> > have dedicated their lives to AppSec and if anyone should win big-time, I
> > hope it's them. I even told Jeff I hope he hits the mother load and
> donates a
> > little back to OWASP.
> >
> >
> >                       However, my instinct and evidence tell me that they
> > both went over the line in the use of the OWASP brand to sell product.
> >
> >
> >                       Now, Jeff makes a good point. We as a board and
> > staff are very poor at enforcing brand management policy and it's not
> fair to
> > single out Contrast, when many other vendors violate the brand, IMO. Just
> > google OWASP and watch the ads fly that use the OWASP name to sell
> > product.
> >
> >
> >                       Also, any and every request that was made of Dave
> to
> > adjust the project for the sake of vendor neutrality was taken very
> seriously.
> > Regardless of Daves past intentions, he is clearly trying to do the
> right thing
> > moving forward.
> >
> >
> >                       I look to "postels principle" in this situation
> (this is
> > otherwise known as the "robustness principle" and dates back to the
> > creation of TCP) . This is paraphrased as, "Be liberal in what you take
> from
> > others but be conservative in what you dish out". So I think it's
> critical that
> > OWASP and any OWASP resource present itself in a strict vendor neutral
> > way. But unless OWASP wants to be much more "even" in the enforcement
> > of brand policy across the board to all violators, we should be fairly
> lax in the
> > enforcement of these issues from the outside world.
> >
> >
> >                       I am trying to be objective here. My trigonometry
> > teacher once told me "I'd fail my mother" when I asked him if he would
> ever
> > fail me (I was an A student). If my mother owned a security company and
> > tried the same stunt, I'd have the same opinions about her actions as
> well.
> >
> >
> >                       So what next? Well hello from the other side. I'm
> > going back to listening to Adele's new album where I can sit in my deep
> > feelings and reflect upon what the OWASP foundation has done to enrich my
> > life. I would much rather keep out of this (and any other conflict laden
> > situation at OWASP), but I feel it's my responsibility to speak up.
> >
> >
> >                       Aloha,
> >
> >
> >
> >                       --
> >
> >                       Jim Manico
> >
> >                       Global Board Member
> >
> >                       OWASP Foundation
> >
> >                       https://www.owasp.org <https://www.owasp.org/>
> >
> >                       Join me in Rome for AppSecEU 2016!
> >
> >
> >                       On Nov 26, 2015, at 9:09 PM, Josh Sokol
> > <josh.sokol at owasp.org> wrote:
> >
> >
> >
> >
> >
> >
> >                               I would be happy to provide an update.
> >
> >
> >
> >
> >                               *       Matt Konda and Dave Wichers, the
> > Benchmark Project Leader, had a conversation a few weeks back.  To
> > summarize their conversation, Dave acknowledges the currently lack of
> > diversity in his project and it is his sincere desire to drive more
> people to it to
> > help.  He also acknowledges the issues with Contrast's extreme marketing
> > around the project and feels that it is in everyone's best interests for
> them to
> > curb it back.  While he does have an ownership stake in Contrast, he
> works at
> > Aspect and has no control over the marketing messages that they are
> putting
> > out there.  From the Board perspective, there has been no evidence of any
> > impropriety on Dave's part and it should be our goal to drive more
> diversity
> > into the project to support Dave.  Dave appears to be sincere in his
> desires to
> > create a tool where OWASP can tell vendors what we expect from their
> > tools.  If the main issue is that only members of Aspect are working on
> it,
> > then the best thing that we can do is try to get him some outside
> assistance.
> > We are also asking that the project be opened up to commits via Git so
> that
> > outsiders can push commits to it.
> >
> >
> >
> >                               *       Josh Sokol and Jeff Williams, the
> CTO
> > of Contrast, had a conversation a few weeks back.  To summarize their
> > conversation, Jeff believes that the work that Dave is doing on the
> > Benchmark is a game changer in that it gives OWASP the power in dictating
> > what these tools need to be finding.  He wants the Benchmark to be
> > successful and understands that it needs to be diverse in order to be
> trusted.
> > He recognizes that Dave is trying to do that and does not want the
> marketing
> > message from Contrast to interfere with his efforts.  Jeff felt that the
> "Lab"
> > status granted to Benchmark meant that it was ready for mainstream
> > adoption, that it had 21k tests, and was almost a year old, and didn't
> see
> > anything wrong with marketing their results, but has agreed to talk to
> their
> > marketing team to get them to lay off that message for now.  From the
> Board
> > perspective, we have come to the realization that our brand usage
> guidelines
> > need an overhaul to clarify what is and is not allowed.  We have made a
> few
> > proposals and have reached out to Mozilla to gain more insight on their
> > guidelines and even ask for assistance.  Noreen and Paul are taking lead
> on
> > these efforts.
> >
> >                               *       There is a note in the notes that
> the
> > Board was supposed to follow up with an open letter to the community and
> > companies involved describing our review and actions.  I don't think
> that has
> > happened so I will remind the person who took on that action item.
> >
> >
> >                               I'm happy to answer any questions that you
> > may have.
> >
> >
> >
> >
> >
> >                               ~josh
> >
> >
> >
> >
> >                               On Thu, Nov 26, 2015 at 11:55 AM, Tobias
> > <tobias.gondrom at owasp.org> wrote:
> >
> >
> >
> >
> >                                       There have been several
> > conversations on that matter and a dedicated call. Unfortunately for
> personal
> > reasons I could not attend the last call as it was at 04:00am my local
> time, but
> > all other board members did participate.
> >
> >
> >
> >                                       Could please one of my fellow board
> > members give an update.
> >
> >
> >
> >                                       Best, Tobias
> >
> >
> >
> >
> >
> >
> >
> >                                       On 26/11/15 18:04, Timo Goosen
> > wrote:
> >
> >
> >
> >
> >                                               I would also like to know
> the
> > answer to Simon's question. We need to get rid of bad apples in OWASP in
> > my opinion, there are too many people just using the OWASP "name" or
> > "brand" to improve their own financial situation or career.
> >
> >
> >                                               Regards.
> >
> >                                               Timo
> >
> >
> >                                               On Thu, Nov 26, 2015 at
> 1:13
> > PM, psiinon <psiinon at gmail.com> wrote:
> >
> >
> >
> >
> >                                                       Paul, and the rest
> of
> > the board,
> >
> >
> >
> >
> >
> >                                                       Its been over 2
> months
> > since I raised this issue.
> >
> >
> >
> >                                                       Whats happening?
> >
> >
> >
> >                                                       Has the board even
> > discussed it?
> >
> >
> >
> >
> >
> >                                                       Cheers,
> >
> >
> >
> >
> >
> >                                                       Simon
> >
> >
> >
> >
> >
> >                                                       On Tue, Oct 20,
> 2015 at
> > 10:00 PM, Paul Ritchie <paul.ritchie at owasp.org> wrote:
> >
> >
> >
> >
> >                                                               Eoin,
> Johanna,
> > All:
> >
> >
> >                                                               In an
> earlier
> > email, Josh Sokol mentioned that he will be speaking in the next day or
> 2 to
> > their CTO, while at LASCON, as a representative of the OWASP Board.
> > Following that feedback, the Board has action to take the next steps.
> >
> >
> >                                                               Just an
> FYI that
> > all comments are recognized and action is being taken.
> >
> >
> >                                                               Paul
> >
> >
> >
> >
> >                                                               Best
> Regards,
> > Paul Ritchie
> >
> >                                                               OWASP
> > Executive Director
> >
> >
> >       paul.ritchie at owasp.org
> >
> >
> >
> >                                                               On Tue,
> Oct 20,
> > 2015 at 1:54 PM, johanna curiel curiel <johanna.curiel at owasp.org> wrote:
> >
> >
> >
> >
> >
>  Time
> > for owasp to do a public statement and put a clear story regarding this
> > abusive behavior of Owasp brand
> >
> >
> >
> >                                                                       On
> > Tuesday, October 20, 2015, Eoin Keary <eoin.keary at owasp.org> wrote:
> >
> >
> >
> >
> >
> >       Folks,
> >
> >
> >
> >       The project should be immediately shelved it's simply bad form.
> >
> >
> >
> >       This is damaging to OWASP, the industry and exactly what OWASP is
> > not about.
> >
> >
> >
> >       There is a clear conflict of interest and distinct lack of science
> behind
> > the claims made by Contrast.
> >
> >
> >
> >
> >
> >
> >
> >
> >       Eoin Keary
> >
> >
> >       OWASP Volunteer
> >
> >
> >       @eoinkeary
> >
> >
> >
> >
> >
> >
> >
> >
> >       On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel
> > <johanna.curiel at owasp.org> wrote:
> >
> >
> >
> >
> >
> >
> >
> >       At the moment we did the project review, we observed that the
> > project did not have enough testing to be considered in any form as
> 'ready'
> > for benchmarking, neither that it had yet the community adoption, however
> > technically speaking as it has been classified by the leaders, the
> project is at
> > the beta stage.
> >
> >
> >
> >       Indeed , Dave had the push to have the project reviewed but it was
> > never clear that later on the project was going to be advertisied this
> way.
> > That all happend after the presentation at Appsec.
> >
> >
> >
> >       I had my concerns regarding how sensitive is the subject of the
> > project ,but I think we should allow project leaders to develop their
> > communication strategy even if this has conflict of interest. It all
> depends
> > how they behave and how they manage this.
> >
> >
> >
> >
> >       On Tuesday, October 6, 2015, Michael Coates
> > <michael.coates at owasp.org> wrote:
> >
> >
> >
> >
> >
> >
> >               It's not really that formal to add to the agenda, just a
> wiki that
> > we add in the text.
> >
> >
> >
> >               I think you can safely assume it will get the appropriate
> > discussion.
> >
> >
> >
> >               On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com>
> > wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                       Really?? Its not on the agenda yet for the next
> > meeting??
> >
> >
> >                       How does it get added to the agenda?
> >
> >
> >
> >
> >
> >                       And that was a formal request if that makes any
> > difference :)
> >
> >
> >
> >
> >
> >                       I'm all in favour of getting the facts straight
> before any
> > actions are taken, hence my request for an 'ethical review' or whatever
> it
> > should be called.
> >
> >
> >
> >
> >
> >
> >
> >
> >                       Cheers,
> >
> >
> >
> >
> >
> >
> >
> >
> >                       Simon
> >
> >
> >
> >
> >
> >
> >                       On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates
> > <michael.coates at owasp.org> wrote:
> >
> >
> >
> >
> >
> >
> >                               First step is to get all of our information
> > straight so we're clear on where things are at.
> >
> >
> >
> >                               This was not on the board agenda last
> > meeting and is also not on the next agenda as of yet (of course it could
> > always be added if needed).
> >
> >
> >
> >                               We are aware that people have raised
> > questions though.   I'm hoping we can get a clear understanding of all
> the
> > facts and then discuss if changes are needed.
> >
> >
> >
> >
> >
> >                               On Oct 6, 2015, at 1:52 AM, psiinon
> > <psiinon at gmail.com> wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                       Hey Michael,
> >
> >
> >
> >
> >
> >                                       Is the board going to take any
> action?
> >
> >
> >
> >
> >
> >                                       Were there any discussions about
> this
> > controversy in the board meeting at AppSec USA?
> >
> >
> >
> >
> >
> >                                       If not will it be on the agenda
> for the
> > meeting on October 14th?
> >
> >
> >
> >
> >
> >
> >
> >
> >                                       Cheers,
> >
> >
> >
> >
> >
> >
> >
> >
> >                                       Simon
> >
> >
> >
> >
> >
> >
> >
> >                                       On Tue, Oct 6, 2015 at 8:25 AM,
> > Michael Coates <michael.coates at owasp.org> wrote:
> >
> >
> >
> >
> >
> >
> >                                               Simon
> >
> >
> >
> >                                               I posted the below message
> > earlier today. At this point my goal is to just gain clarity over the
> current
> > reality and ideally drive to a shared state of success. This message
> doesn't
> > seem to be reflected in the list yet. It could be because my membership
> > hasn't been approved or because of mail list delays (I miss Google
> groups).
> > But I think these questions will start the conversation.
> >
> >
> >
> >                                               (This was just me asking
> > questions as a curious Owasp member, not any action on behalf of the
> > board)
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                               Begin forwarded message:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       From: Michael
> Coates
> > <michael.coates at owasp.org>
> >
> >
> >                                                       Date: October 5,
> 2015
> > at 6:20:23 PM PDT
> >
> >
> >                                                       To:
> owasp-benchmark-
> > project at lists.owasp.org
> >
> >
> >                                                       Subject: Project
> > Questions
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       OWASP Benchmark
> > List,
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       I've heard more
> about
> > this project and am excited about the idea of an independent perspective
> of
> > tool performance. I'm trying to understand a few things to better
> respond to
> > questions from those in the security & OWASP community.
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       In my mind there
> are
> > two big areas for consideration in a benchmark process.
> >
> >
> >                                                       1. Are the
> benchmarks
> > testing the right areas?
> >
> >
> >                                                       2. Is the process
> for
> > creating the benchmark objective & free from conflicts of interest.
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       I think as a group
> > OWASP is the right body to align on #1.
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       I'd like to ask
> for some
> > clarifications on item #2. I think it's important to avoid actual
> conflict of
> > interest and also the appearance of conflict of interest. The former is
> obvious
> > why we mustn't have that, the latter is critical so others have faith in
> the tool,
> > process and outputs of the process when viewing or hearing about the
> > project.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       1) Can we clarify
> > whether other individuals have submitted meaningful code to the project?
> >
> >
> >                                                       Observation:
> >
> >
> >                                                       Nearly all the code
> > commits have come from 1 person (project lead).
> >
> >
> >
> >       https://github.com/OWASP/Benchmark/graphs/contributors
> > <https://github.com/OWASP/Benchmark/graphs/contributors>
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       2) Can we clarify
> the
> > contributions of others and their represented organizations?
> >
> >
> >                                                       Observation:
> >
> >
> >                                                       The
> > acknowledgements tab listed two developers (Juan Gama & Nick Sanidas)
> > both who work at the same company as the project lead. It seems other
> > people have submitted some small amounts of material, but overall it
> seems
> > all development has come from the same company.
> >
> >
> >
> >       https://www.owasp.org/index.php/Benchmark#tab=Acknowledgem
> > ents
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       3) Can we clarify
> in
> > what ways we've mitigated the potential conflict of interest and also the
> > appearance of a conflict of interest? This seems like the largest
> blocker for
> > wide spread acceptance of this project and the biggest risk.
> >
> >
> >                                                       Observation:
> >
> >
> >                                                       The project lead
> and
> > both of the project developers works for a company with very close ties
> to
> > one of the companies that is evaluated by this project. Further, it
> appears
> > the company is performing very well on the project tests.
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       4) If we are going
> to list
> > tool vendors then I'd recommend listing multiple vendors for each
> category.
> >
> >
> >                                                       Observation:
> >
> >
> >                                                       The tools page only
> > lists 1 IAST tool. Since this is the point of the potential conflict of
> interest it is
> > important to list numerous IAST tools.
> >
> >
> >
> >       https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_
> > 2FResults
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       5) Diverse body
> with
> > multiple points of view
> >
> >
> >                                                       Observation:
> >
> >
> >                                                       There is no
> indication
> > that multiple stakeholders are present to review and decide on the
> future of
> > this project. If they exist, a new section should be added to the
> project page
> > to raise awareness. If they don't exist, we should reevaluate how we are
> > obtaining an independent view of the testing process.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       Again, I think the
> idea
> > of the project is great. From my perspective clarifying these questions
> will
> > help ensure the project is not only objective, but also perceived as
> objective
> > from someone reviewing the material. Ultimately this will contribute to
> the
> > success and growth of the project.
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       Thanks!
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       --
> >
> >
> >                                                       Michael Coates
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                               On Oct 2, 2015, at 1:31 AM,
> > psiinon <psiinon at gmail.com> wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       OK, based on the
> > concerns raised so far I think the board should initiate a review of the
> OWASP
> > Benchmark project.
> >
> >
> >
> >
> >
> >                                                       I'm not raising a
> formal
> > complaint against it, I'm just requesting a review.
> >
> >
> >
> >
> >
> >                                                       And I dont think it
> > needs a 'standard' project review - Johanna has already done a very good
> job
> > of this.
> >
> >
> >
> >
> >
> >                                                       Not sure what sort
> of
> > review you'd call it, I'll leave the naming to others :)
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       I'm concerned that
> we
> > have an OWASP project lead by a company who has a clear commercial stake
> > in the results.
> >
> >
> >
> >
> >
> >                                                       Bringing more
> > companies on board will help, but I'm still not sure that alone will
> make it
> > independent enough.
> >
> >
> >
> >
> >
> >                                                       Commercial
> companies
> > can afford to dedicate staff to improving Benchmark so that their
> products
> > look better.
> >
> >
> >
> >
> >
> >                                                       Open source
> projects
> > just cant do that, so we are at a distinct disadvantage.
> >
> >
> >
> >
> >
> >                                                       Should we allow a
> > commercially driven OWASP project who's aim could be seen be to promote
> > commercial software?
> >
> >
> >
> >
> >
> >                                                       If so, what sort of
> > checks and balances does it need?
> >
> >
> >
> >
> >
> >                                                       Those are the sort
> of
> > questions I'd like an independent review to look at.
> >
> >
> >
> >
> >
> >
> >                                                       I do think there
> are
> > some immediate steps that could be taken:
> >
> >
> >
> >
> >
> >
> >                                                       *       I'd like
> to see
> > the Benchmark project page clearly state thats its at a very early stage
> and
> > that the results are _not_ yet suitable for use in commercial literature.
> >
> >
> >                                                       *       I'd also
> like the
> > main companies developing Benchmark to be clearly stated on the main
> > page. If and when other companies get involved then this would actually
> > help the project's claim of vendor independence.
> >
> >
> >                                                       *       And I'd
> love to
> > see a respected co-leader added to the project who is not associated with
> > any commercial or open source security tools:)
> >
> >
> >
> >                                                       And we should carry
> > on discussing the project on this list - I think such discussions are
> very
> > healthy, and I'd love to see this project mature to a state where it can
> be a
> > trusted, independent and valued resource.
> >
> >
> >
> >
> >
> >
> >                                                       Cheers,
> >
> >
> >
> >
> >
> >
> >
> >
> >                                                       Simon
> >
> >
> >
> >
> >
> >
> >                                                       On Thu, Oct 1,
> 2015 at
> > 7:59 PM, Tobias <tobias.gondrom at owasp.org> wrote:
> >
> >
> >
> >
> >
> >
> >                                                               @Simon:
> >
> >
> >                                                               yes, the
> > leaders list is the place for your discussions for project and chapter
> leaders
> >
> >
> >                                                               @Timo: I
> like
> > your framing of "Don't ask what OWASP can do for me, ask what I can do
> for
> > OWASP."
> >
> >
> >                                                               That should
> > and is indeed the spirit of OWASP:-)
> >
> >
> >                                                               Best
> regards,
> > Tobias
> >
> >
> >
> >
> >
> >
> >                                                               On 30/09/15
> > 09:42, Timo Goosen wrote:
> >
> >
> >
> >
> >       ...
> >
> >       [Message clipped]
> >       _______________________________________________
> >       Owasp-board mailing list
> >       Owasp-board at lists.owasp.org
> >       https://lists.owasp.org/mailman/listinfo/owasp-board
> >
> >
> >
> >
> >
> >
> > --
> >
> > OWASP ZAP <https://www.owasp.org/index.php/ZAP>  Project leader
>
>
>
>
> _______________________________________________
> OWASP-Leaders mailing list
> OWASP-Leaders at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>



-- 


Claudia Aviles-Casanovas <claudia.aviles-casanovas at owasp.org>
Project Coordinator
Phone:973-288-1697
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151130/939ba5c2/attachment-0001.html>


More information about the OWASP-Leaders mailing list