[Owasp-leaders] [Owasp-board] OWASP Benchmark project - potential conflict of interest

johanna curiel curiel johanna.curiel at owasp.org
Fri Nov 27 21:02:36 UTC 2015

>>But yes, the takeaway is exactly as you say, we need to learn from this
experience, improve our policies and guidelines, and move forward.

I don't think we can just move 'fwd'.

I think OWASP needs to make a statement publicly as son as possible that
OWASP does not agree how this was/is handled by Contrast.

Right now, if there is no statements regarding which position OWASP has
taken , this looks like Jim said
" brush it under the rug."

When people abuse of the brand like they did (not just a logo or talk
commercially in a chapter meeting which is insignificant to this abuse)
there should be consequences, otherwise people will keep on abusing and use
every opportunity to do so.

Jeff tried also brushing the dirt under the rug by trying to convince you
Josh that it's all the machiavellian mind' of Contrast marketing
people...infringement number 2. It seems you were not aware at all of all
the twitter and media circus around OWASP name brand abuse and benchmark

You spoke with him some weeks ago? On 12 November Jeff was doing marketing
on twitter with benchmark....this video should take also down with Jeff
speaking, OWASP logo and DHS logo!


[image: Inline image 1]

 I will repeat the words done by Chris Wysopal in his article as conclusion
to the Benchmark project

Once developed, the OWASP benchmark has the potential to be a valuable tool
for companies struggling with application security challenges. I*n its
current state, it does everyone a disservice by indicating there’s a silver
bullet to solve their problems.* *We all know this to be a silly
presumption.* Application security is a difficult challenge and businesses
need help to understand which techniques can and should be applied to the
various aspects of the challenge. *It’s time for the vendor community to
stop being salesmen and start being advisors on a complicated issue.*
Software has eaten the world and attackers are gorging themselves on a
seemingly endless supply of vulnerabilities.

While I think the same as him, this project should be set back to Incubator
as a punishment and a statement.

*Tony Turner>>Lastly, it is my opinion that project leadership should not
belong to anyone working for or with a partnership/ownership stake for any
vendor being evaluated. I think this is a flawed model and should
transition to a vendor neutral party.*

Allowing this project to exist without being properly tested with other
vendor tools by a group that has no conflict of interest is also a very bad
message OWASP sends.

Thats why I set a project research for this project:

Even worse this project received sponsorship of DHS . How much? I'm

You know how bad this looks? i hope so this makes you more aware, and
actions are taken not just 'change our guidelines'...

@Tony: I'm looking to do an independent research and publish my findings. I
have no commercial involvement in this project and I believe I can provide
neutral view of the effectiveness of the tool, what does cover and what not

If OWASP and the vendor community supports my research , then we are
talking about a neutral benchmark project.



On Fri, Nov 27, 2015 at 4:19 PM, Josh Sokol <josh.sokol at owasp.org> wrote:

> I agree with the assertion that Contrast was using OWASP Benchmark as a
> marketing vehicle.  I agree that it is not something that OWASP should
> allow.  But our Brand Usage Guideliness are not very clear on what is and
> is not appropriate and they need updates to make it clear.  Paul and Noreen
> are working on that.  I'm not really sure if I draw the conclusion that
> there is abuse on behalf of our project lead.  Just speculation there.  But
> yes, the takeaway is exactly as you say, we need to learn from this
> experience, improve our policies and guidelines, and move forward.
> ~josh
> On Fri, Nov 27, 2015 at 1:43 PM, Eoin Keary <eoin.keary at owasp.org> wrote:
>> I suppose bottom line is many in the industry thought it looked pretty
>> shite. Many also feel it was using OWASP as a Marketing vehicle. The
>> problem is what defines an OWASP project? Are there governance controls to
>> help prevent "abuse" of a project by project leads? Some times it not what
>> is said but how it's delivered.
>> But we should move on and learn from the experience.
>> Eoin Keary
>> OWASP Volunteer
>> @eoinkeary
>> On 27 Nov 2015, at 6:47 p.m., Josh Sokol <josh.sokol at owasp.org> wrote:
>> Thank you for the links to those articles.  The first one discusses the
>> strengths and weaknesses of the different methods of evaluating for
>> application vulnerabilities.  The section on the Benchmark seems wholly
>> appropriate to me.  That seems like an excellent description of what the
>> project is designed to do.  I see some metrics in there about which tools
>> are more effective on which types of vulnerabilities, but I don't see him
>> straight up saying "The OWASP Benchmark proves that Contrast is better".
>> This seems like statements made based on some level of testing and
>> research.  Honestly, I don't see any OWASP brand abuse in that article.
>> Whether it's in good taste or not at this stage in the project is certainly
>> debatable, but if you look at the brand usage guidelines (
>> https://www.owasp.org/index.php/Marketing/Resources#tab=BRAND_GUIDELINES),
>> I don't see any violations.  We need to govern to policy here which is why
>> Paul and Noreen are evaluating changes to the guidelines and our
>> enforcement policies to make abuse more difficult.
>> The second article is a competing vendor's reaction to the first.  He
>> makes some good points about the issues with Benchmark, but he also says
>> that he hopes that it will be improved over time, and Dave has committed to
>> that.  What I don't see is the vendor saying "...and Veracode has committed
>> resources to help make the Benchmark more accurate across all tool sets".
>> The Benchmark page is pretty clear that it does it's best to provide a
>> benchmark without working exactly like a real-world application.  Maybe
>> some more disclaimer text about where the project is at today would be in
>> order to validate some of Chris' concerns, but I hardly see this as "brand
>> abuse" or a reason to demote the project.
>> Please consider that I have spoken with both Dave and Jeff on this topic
>> and read much of the discussions around it before formulating my opinion.
>> I doubt that you have done the same so I'm not sure how you can claim that
>> you have researched the issues and all parties involved when you haven't
>> even spoken with the two people whom you are accusing of impropriety.  I
>> have no bias here.  I am simply speaking with the individuals involved,
>> looking at the currently OWASP policies and guidelines, and helping to
>> determine our next steps.
>> ~josh
>> On Fri, Nov 27, 2015 at 12:22 PM, johanna curiel curiel <
>> johanna.curiel at owasp.org> wrote:
>>> >>While I agree with you that there has been some brand abuse, it was
>>> abuse by Contrast (specifically their marketing department), and not by
>>> "these gentlemen" as  you state.
>>> Really? ..'some brand abuse'..this is more than brand abuse
>>> Josh , please read also the article written by Jeff
>>> http://www.darkreading.com/vulnerabilities---threats/why-its-insane-to-trust-static-analysis/a/d-id/1322274
>>> ?
>>> And Veracode's reaction including others in Twitter
>>> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>> My strong advice is to research the issues and all the parties involved
>>> before making statements
>>> On Fri, Nov 27, 2015 at 2:07 PM, Josh Sokol <josh.sokol at owasp.org>
>>> wrote:
>>>> Jim,
>>>> A concern was expressed to the Board and, frankly, I am insulted by you
>>>> saying that this was "brushed under the rug".  The Board delegated Matt to
>>>> talk with Dave and they had a lengthy conversation on the subject.  The
>>>> Board delegated me to talk with Jeff and we had a lengthy conversation on
>>>> the subject.  If you do not trust in our abilities to read people, ask the
>>>> right questions, and provide honest feedback about our conversations, then
>>>> that's a bigger issue that we should take offline.  After our
>>>> conversations, we took the time to call a special two-hour session of the
>>>> Board in order to discuss this subject (and only this subject).  We spoke
>>>> about all facets of the issue at hand, about the challenges and possible
>>>> solutions, and concluded on some very concrete next steps.
>>>> While I agree with you that there has been some brand abuse, it was
>>>> abuse by Contrast (specifically their marketing department), and not by
>>>> "these gentlemen" as  you state.  Unless you can point to some sort of
>>>> evidence showing that Jeff and/or Dave first-hand abused the brand, then I
>>>> believe that you are speaking with your heart instead of with your head.  I
>>>> appreciate your passion, but I label this as conspiracy theory because
>>>> without evidence to support your claims, I cannot accept it as anything
>>>> other.
>>>> ~josh
>>>> On Fri, Nov 27, 2015 at 11:39 AM, Jim Manico <jim.manico at owasp.org>
>>>> wrote:
>>>>> Josh,
>>>>> I stand by my comments and perspective, but I'm disheartened that you
>>>>> consider my presentation of facts (and the concerns of many active members
>>>>> of our community) as a "conspiracy theory".
>>>>> In my experience, these kind of comments border on insults and only
>>>>> cause folks to harden their opinions.
>>>>> Once again I feel these gentlemen got away with a kind of brand abuse
>>>>> that is very hurtful to the OWASP community but I am at a loss as to how
>>>>> handle or prevent these kinds of mishaps - especially when board members
>>>>> like yourself seem willing to - from what I see - brush it under the rug.
>>>>> --
>>>>> Jim Manico
>>>>> Global Board Member
>>>>> OWASP Foundation
>>>>> https://www.owasp.org
>>>>> Join me in Rome for AppSecEU 2016!
>>>>> On Nov 27, 2015, at 7:23 PM, Josh Sokol <josh.sokol at owasp.org> wrote:
>>>>> Admittedly, this was my gut reaction at first as well.  I began
>>>>> linking all of these companies, people, and projects together in my mind
>>>>> (there are some loose links there) and painted a big conspiracy picture
>>>>> similar to what Jim and Dinis have stated.  But, after speaking directly
>>>>> with Jeff, and hearing about the conversation that Dave and Matt had, I've
>>>>> changed my mind.
>>>>> I think it begins with the project itself.  If you aren't sold on the
>>>>> idea of the Benchmark, then you'll never be able to get to the same place.
>>>>> My original line of thinking was that it was just a bar for vendors to
>>>>> compare their tools against eachother, but that's a bit myopic.  We are in
>>>>> an industry where things evolve very quickly.  As a customer of these
>>>>> tools, I know firsthand that something that a tool does today may not be
>>>>> the case a week from now.  Likewise, new features are being added daily and
>>>>> I need a point-in-time metric to be able to gauge continual effectiveness.
>>>>> Cool, right?  But not a game changer.  The game changer part comes when you
>>>>> realize that by developing and evolving the tests that go into the
>>>>> Benchmark, we are moving the bar higher and higher.  We (OWASP) are
>>>>> effectively setting the standard by which these tools will be compared.  A
>>>>> tool that receives a lower score on the Benchmark today knows exactly what
>>>>> they need to work on in order to pass that test tomorrow and we already
>>>>> have examples of tools that have made improvements because of their
>>>>> Benchmark score (Ask Simon about ZAP's experience with the Benchmark).  I
>>>>> don't think that anyone can argue that the Benchmark project isn't being
>>>>> effective when OWASP's own tools are being driven forward as a result of
>>>>> using it.
>>>>> But, but, but, Dave and Jeff own Aspect and have stock in Contrast and
>>>>> Jeff is the Contrast CTO and Contrast got good scores so it's a conspiracy
>>>>> right?  Is there some code that allows Contrast to use the Benchmark?
>>>>> Absolutely.  Can you really blame Dave for starting his testing on the
>>>>> effectiveness of the Benchmark with a tool that he owned and is familiar
>>>>> with?  If I were going to start a similar project, there's no question in
>>>>> my mind that I would begin my testing with the tools that I have available
>>>>> to me.  That said, is there code that allows other tools to use the
>>>>> Benchmark?  Absolutely.
>>>>> Regarding "Dave has a history of breaching his duty to be vendor
>>>>> neutral", while I cannot comment on his past actions, I can judge what
>>>>> we've seen recently.  Matt saw a presentation from Dave on the Benchmark at
>>>>> a conference in Chicago.  He said that he felt that the message was
>>>>> appropriate and while IAST tools were mentioned as receiving higher scores,
>>>>> it wasn't a "Contrast is the best" type of message, more of a generality.
>>>>> I saw a very similar (if not the same) talk by Jeff at LASCON 2015 and the
>>>>> message was exactly the same.  I watched the talk expecting some sort of
>>>>> impropriety, but found none.  So, perhaps Dave has abused some privilege
>>>>> granted to him in the past, but what I've seen from him at this point, with
>>>>> respect to the Benchmark, has been appropriate.
>>>>> You have a very good point with respect to the Contrast marketing
>>>>> message around the Benchmark.  It's been completely absurd, over the top,
>>>>> and, in my personal opinion, intolerable.  In fact, I experienced the same
>>>>> thing that you talked about with them at LASCON 2015 where they stood in
>>>>> front of the door of the room Jeff was speaking in and scanned attendees as
>>>>> they went into the talk.  I agree that these types of aggressive marketing
>>>>> tactics cannot be tolerated at OWASP.  In addition, we have seen several
>>>>> marketing messages from them effectively implying that OWASP endorses
>>>>> Contrast.  Clearly this is not OK.  I've spoken with Jeff about it and we
>>>>> agreed that it is not in the Benchmark's best interest to have this
>>>>> aggressive Contrast marketing around it at such an early stage.  He has
>>>>> said that he is not responsible for Contrast's marketing team, but that he
>>>>> would speak with the people who are.  I haven't seen a single message from
>>>>> them since so I'm guessing that he's made good on this promise.  While
>>>>> that's an excellent start, OWASP's takeaway here should be that we need to
>>>>> do a better job with our brand usage guidelines both in terms of the
>>>>> wording and enforcement.  There are many other companies out there that use
>>>>> the OWASP brand and I think that we agree that selective enforcement
>>>>> against Contrast is not the right answer.  Paul and Noreen are actively
>>>>> working on this.  Either way, I think that implying that activities from a
>>>>> vendor's marketing department means that the project is not objective is
>>>>> not inappropriate.  If we feel that the project is not objective, then
>>>>> separate measures need to be taken to drive contribution diversity into
>>>>> it.  That I absolutely agree with and the message from Dave was that he
>>>>> would love to have more contributors to his project.  But, seeing as we
>>>>> cannot force people to work on it, this becomes a matter of "put up or shut
>>>>> up".  The same goes for the experts that you said reviewed the code.  If
>>>>> they feel that it is somehow skewed towards Contrast, they have the power
>>>>> to change that.  Now, if someone tries to participate and Dave tells them
>>>>> "No thanks", then I agree we have a problem, but I don't hear anyone
>>>>> inferring that happened.
>>>>> Please, let's drop the conspiracy theories and focus on the tangible
>>>>> things that we can do to help an OWASP project to be more successful.  Help
>>>>> find more participants to drive diversity, update our brand usage
>>>>> guidelines to prevent abuse, enforce them widely, etc.  Thank you.
>>>>> ~josh
>>>>> On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico <jim.manico at owasp.org>
>>>>> wrote:
>>>>>> Dinis,
>>>>>> Like a rare celestial moment when all the planets plus Pluto are
>>>>>> aligned, I just read your email on the future of OWASP projects thinking,
>>>>>> "Dinis is spot on".
>>>>>> Reflecting on projects I manage or work on...
>>>>>> The Java Encoder and HTML Sanitizer are likely best moved to Apache
>>>>>> now that they have reached a measure of adoption and maturity. Apache would
>>>>>> be a much better long term custodian. Perhaps the same for AppSensor, but
>>>>>> not my project - just thinking out loud.
>>>>>> Other similar defensive projects are still being noodled on, so OWASP
>>>>>> is a decent home for these research efforts.
>>>>>> The whole tools category is also something to consider. Dependency
>>>>>> Check and of course ZAP are some of the best projects that OWASP offers,
>>>>>> are they best served where they are today? Both have rich communities of
>>>>>> developers but I don't see the foundation doing much to support these
>>>>>> efforts.
>>>>>> ASVS has the opportunity to effect massive change, I would to love to
>>>>>> see major investment and volunteer activity here. Pro tech writer, detailed
>>>>>> discourses on each individual requirement, etc. If I was king (and I am
>>>>>> not, at all) I would invest in ASVS on a 6 figure scale. (And who started
>>>>>> ASVS? Jeff, Dave and Boberski, hat tip to such a marvelous idea). Or maybe
>>>>>> moving ASVS to the W3C or IETF would help it grow?
>>>>>> The Proactive Controls was a pet project but as we approach 2.0 we
>>>>>> have several active/awesome volunteers working on it. We will be making the
>>>>>> doc "world editable" to make contributions easy. OWASP seems like a good
>>>>>> home for such an awareness doc. Same with T10, especially if community
>>>>>> edits are welcome.
>>>>>> Anyhow, I'm with you on this Dinis. Once a project starts to reach
>>>>>> production quality, spinning off the project as an external project or
>>>>>> moving it to a different foundation where managing production software or
>>>>>> formal standards is their thing seems realistic.
>>>>>> I don't have all the answers here, but your email certainly resonated
>>>>>> with me.
>>>>>> Aloha,
>>>>>> --
>>>>>> Jim Manico
>>>>>> Global Board Member
>>>>>> OWASP Foundation
>>>>>> https://www.owasp.org
>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>> On Nov 26, 2015, at 11:26 PM, Dinis Cruz <dinis.cruz at owasp.org>
>>>>>> wrote:
>>>>>> Jim's reading of this situation is exactly my view on the value of
>>>>>> the Contrast tool and how it has been 'pushing' the rules of engagement to
>>>>>> an very 'fuzzy' moral/ethical/commercial limit :)
>>>>>> As per my last email, a key problem here is the 'perceived
>>>>>> expectation' of what is an OWASP project, and how it should be consumed.
>>>>>> If you look at the OWASP benchmark as a research project, then the
>>>>>> only way it could be making the kind of claims it makes (and have
>>>>>> credibility) is if it had evolved from OWASP, with its own (diverse)
>>>>>> community
>>>>>> On 26 November 2015 at 21:01, Jim Manico <jim.manico at owasp.org>
>>>>>> wrote:
>>>>>>> I have a different take on this situation but my opinion is the
>>>>>>> "minority opinion". I will respect the rest of the boards take on this, but
>>>>>>> here is how I see it.
>>>>>>> First of all, Jeff has stated that he feels I am attacking him
>>>>>>> personally from a past personal grudge, and frankly I do not fault him for
>>>>>>> that perspective since we definitely have history with conflict. So it's
>>>>>>> fair to take my opinion on this with a grain of salt.
>>>>>>> I look at this situation from the perspective of a forensic
>>>>>>> investigator.
>>>>>>> 1) The Benchmark project had Contrast hooks and only Contrast hooks
>>>>>>> in it when I reviewed it so this leads me to believe that the project was
>>>>>>> clearly built with Contrast in mind from the ground up, at least in some
>>>>>>> way.
>>>>>>> 3) Dave has a history of breaching his duty to be vendor neutral. He
>>>>>>> was gifted with a keynote in South Korea a few years ago, and used that
>>>>>>> opportunity to discuss and pitch Contrast, on stage, during a keynote -
>>>>>>> with Contrast specific slides. This is just supporting evidence of his
>>>>>>> intention at OWASP to push Contrast in ways that I think are against the
>>>>>>> intentions and goals of our foundation.
>>>>>>> 3) Other experts have reviewed the project and felt that many of the
>>>>>>> tests were very slanted and almost contrived to support Contrast. I can
>>>>>>> drag those folks into this conversation, but I do not think that would help
>>>>>>> in any way. So it's fair to call this point heresy.
>>>>>>> 4) I do not see this project as revolutionary, at all. Every vendor
>>>>>>> has their own test suite tuned for their tool. As the benchmark stands
>>>>>>> today, I see it as just another vendors product-specific benchmark. Mass
>>>>>>> collaboration from many vendors is not just a "nice to have" but a base
>>>>>>> requirement to get even close to useful for objective tool measurement.
>>>>>>> 5) Jeff stating that his Marketing people went over the line is also
>>>>>>> an admission that - well, they went over the line. By the same token Jeff
>>>>>>> was in his booth at AppSec USA surrounded by benchmark marketing material,
>>>>>>> discussing this to prospects and he even asked me and Mr Coates to wade
>>>>>>> into this debate and support Dave. So to say he was not involved and it was
>>>>>>> only his marketing people seems a stretch at best.
>>>>>>> 6) The Contrast marketing team was wandering around the conference
>>>>>>> zapping folks to get leads, and I asked them to stay in their booth, which
>>>>>>> is standard conference policy. These folks know better but are again going
>>>>>>> over the line to sell product at OWASP. There is a better way (like
>>>>>>> focusing on product capability and language support, have consistent +
>>>>>>> stellar customer service, have a humble and gracious attitude to all
>>>>>>> prospects and customers, actively participate in OWASP in a vendor neutral
>>>>>>> and community supportive way, etc).
>>>>>>> Please note, I think Contrast is a decent tool, I've offered to
>>>>>>> resell in the past, and I have recommended it in certain situations - even
>>>>>>> after this situation arose. I'm stating this out of honestly and desire to
>>>>>>> put my cards on the table. I truly want Jeff and Dave to be successful.
>>>>>>> They have dedicated their lives to AppSec and if anyone should win
>>>>>>> big-time, I hope it's them. I even told Jeff I hope he hits the mother load
>>>>>>> and donates a little back to OWASP.
>>>>>>> However, my instinct and evidence tell me that they both went over
>>>>>>> the line in the use of the OWASP brand to sell product.
>>>>>>> Now, Jeff makes a good point. We as a board and staff are very poor
>>>>>>> at enforcing brand management policy and it's not fair to single out
>>>>>>> Contrast, when many other vendors violate the brand, IMO. Just google OWASP
>>>>>>> and watch the ads fly that use the OWASP name to sell product.
>>>>>>> Also, any and every request that was made of Dave to adjust the
>>>>>>> project for the sake of vendor neutrality was taken very seriously.
>>>>>>> Regardless of Daves past intentions, he is clearly trying to do the right
>>>>>>> thing moving forward.
>>>>>>> I look to "postels principle" in this situation (this is otherwise
>>>>>>> known as the "robustness principle" and dates back to the creation of TCP)
>>>>>>> . This is paraphrased as, "Be liberal in what you take from others but be
>>>>>>> conservative in what you dish out". So I think it's critical that OWASP and
>>>>>>> any OWASP resource present itself in a strict vendor neutral way. But
>>>>>>> unless OWASP wants to be much more "even" in the enforcement of brand
>>>>>>> policy across the board to all violators, we should be fairly lax in the
>>>>>>> enforcement of these issues from the outside world.
>>>>>>> I am trying to be objective here. My trigonometry teacher once told
>>>>>>> me "I'd fail my mother" when I asked him if he would ever fail me (I was an
>>>>>>> A student). If my mother owned a security company and tried the same stunt,
>>>>>>> I'd have the same opinions about her actions as well.
>>>>>>> So what next? Well hello from the other side. I'm going back to
>>>>>>> listening to Adele's new album where I can sit in my deep feelings and
>>>>>>> reflect upon what the OWASP foundation has done to enrich my life. I would
>>>>>>> much rather keep out of this (and any other conflict laden situation at
>>>>>>> OWASP), but I feel it's my responsibility to speak up.
>>>>>>> Aloha,
>>>>>>> --
>>>>>>> Jim Manico
>>>>>>> Global Board Member
>>>>>>> OWASP Foundation
>>>>>>> https://www.owasp.org
>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>> On Nov 26, 2015, at 9:09 PM, Josh Sokol <josh.sokol at owasp.org>
>>>>>>> wrote:
>>>>>>> I would be happy to provide an update.
>>>>>>>    - Matt Konda and Dave Wichers, the Benchmark Project Leader, had
>>>>>>>    a conversation a few weeks back.  To summarize their conversation, Dave
>>>>>>>    acknowledges the currently lack of diversity in his project and it is his
>>>>>>>    sincere desire to drive more people to it to help.  He also acknowledges
>>>>>>>    the issues with Contrast's extreme marketing around the project and feels
>>>>>>>    that it is in everyone's best interests for them to curb it back.  While he
>>>>>>>    does have an ownership stake in Contrast, he works at Aspect and has no
>>>>>>>    control over the marketing messages that they are putting out there.  From
>>>>>>>    the Board perspective, there has been no evidence of any impropriety on
>>>>>>>    Dave's part and it should be our goal to drive more diversity into the
>>>>>>>    project to support Dave.  Dave appears to be sincere in his desires to
>>>>>>>    create a tool where OWASP can tell vendors what we expect from their
>>>>>>>    tools.  If the main issue is that only members of Aspect are working on it,
>>>>>>>    then the best thing that we can do is try to get him some outside
>>>>>>>    assistance.  We are also asking that the project be opened up to commits
>>>>>>>    via Git so that outsiders can push commits to it.
>>>>>>>    - Josh Sokol and Jeff Williams, the CTO of Contrast, had a
>>>>>>>    conversation a few weeks back.  To summarize their conversation, Jeff
>>>>>>>    believes that the work that Dave is doing on the Benchmark is a game
>>>>>>>    changer in that it gives OWASP the power in dictating what these tools need
>>>>>>>    to be finding.  He wants the Benchmark to be successful and understands
>>>>>>>    that it needs to be diverse in order to be trusted.  He recognizes that
>>>>>>>    Dave is trying to do that and does not want the marketing message from
>>>>>>>    Contrast to interfere with his efforts.  Jeff felt that the "Lab" status
>>>>>>>    granted to Benchmark meant that it was ready for mainstream adoption, that
>>>>>>>    it had 21k tests, and was almost a year old, and didn't see anything wrong
>>>>>>>    with marketing their results, but has agreed to talk to their marketing
>>>>>>>    team to get them to lay off that message for now.  From the Board
>>>>>>>    perspective, we have come to the realization that our brand usage
>>>>>>>    guidelines need an overhaul to clarify what is and is not allowed.  We have
>>>>>>>    made a few proposals and have reached out to Mozilla to gain more insight
>>>>>>>    on their guidelines and even ask for assistance.  Noreen and Paul are
>>>>>>>    taking lead on these efforts.
>>>>>>>    - There is a note in the notes that the Board was supposed to
>>>>>>>    follow up with an open letter to the community and companies involved
>>>>>>>    describing our review and actions.  I don't think that has happened so I
>>>>>>>    will remind the person who took on that action item.
>>>>>>> I'm happy to answer any questions that you may have.
>>>>>>> ~josh
>>>>>>> On Thu, Nov 26, 2015 at 11:55 AM, Tobias <tobias.gondrom at owasp.org>
>>>>>>> wrote:
>>>>>>>> There have been several conversations on that matter and a
>>>>>>>> dedicated call. Unfortunately for personal reasons I could not attend the
>>>>>>>> last call as it was at 04:00am my local time, but all other board members
>>>>>>>> did participate.
>>>>>>>> Could please one of my fellow board members give an update.
>>>>>>>> Best, Tobias
>>>>>>>> On 26/11/15 18:04, Timo Goosen wrote:
>>>>>>>> I would also like to know the answer to Simon's question. We need
>>>>>>>> to get rid of bad apples in OWASP in my opinion, there are too many people
>>>>>>>> just using the OWASP "name" or "brand" to improve their own financial
>>>>>>>> situation or career.
>>>>>>>> Regards.
>>>>>>>> Timo
>>>>>>>> On Thu, Nov 26, 2015 at 1:13 PM, psiinon <psiinon at gmail.com> wrote:
>>>>>>>>> Paul, and the rest of the board,
>>>>>>>>> Its been over 2 months since I raised this issue.
>>>>>>>>> Whats happening?
>>>>>>>>> Has the board even discussed it?
>>>>>>>>> Cheers,
>>>>>>>>> Simon
>>>>>>>>> On Tue, Oct 20, 2015 at 10:00 PM, Paul Ritchie <
>>>>>>>>> paul.ritchie at owasp.org> wrote:
>>>>>>>>>> Eoin, Johanna, All:
>>>>>>>>>> In an earlier email, Josh Sokol mentioned that he will be
>>>>>>>>>> speaking in the next day or 2 to their CTO, while at LASCON, as a
>>>>>>>>>> representative of the OWASP Board.  Following that feedback, the Board has
>>>>>>>>>> action to take the next steps.
>>>>>>>>>> Just an FYI that all comments are recognized and action is being
>>>>>>>>>> taken.
>>>>>>>>>> Paul
>>>>>>>>>> Best Regards, Paul Ritchie
>>>>>>>>>> OWASP Executive Director
>>>>>>>>>> paul.ritchie at owasp.org
>>>>>>>>>> On Tue, Oct 20, 2015 at 1:54 PM, johanna curiel curiel <
>>>>>>>>>> johanna.curiel at owasp.org> wrote:
>>>>>>>>>>> Time for owasp to do a public statement and put a clear story
>>>>>>>>>>> regarding this abusive behavior of Owasp brand
>>>>>>>>>>> On Tuesday, October 20, 2015, Eoin Keary <eoin.keary at owasp.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Folks,
>>>>>>>>>>>> The project should be immediately shelved it's simply bad form.
>>>>>>>>>>>> This is damaging to OWASP, the industry and exactly what OWASP
>>>>>>>>>>>> is not about.
>>>>>>>>>>>> There is a clear conflict of interest and distinct lack of
>>>>>>>>>>>> science behind the claims made by Contrast.
>>>>>>>>>>>> Eoin Keary
>>>>>>>>>>>> OWASP Volunteer
>>>>>>>>>>>> @eoinkeary
>>>>>>>>>>>> On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel <
>>>>>>>>>>>> johanna.curiel at owasp.org> wrote:
>>>>>>>>>>>> At the moment we did the project review, we observed that the
>>>>>>>>>>>> project did not have enough testing to be considered in any form as 'ready'
>>>>>>>>>>>>  for benchmarking, neither that it had yet the community adoption, however
>>>>>>>>>>>> technically speaking as it has been classified by the leaders, the project
>>>>>>>>>>>> is at the beta stage.
>>>>>>>>>>>> Indeed , Dave had the push to have the project reviewed but it
>>>>>>>>>>>> was never clear that later on the project was going to be advertisied this
>>>>>>>>>>>> way. That all happend after the presentation at Appsec.
>>>>>>>>>>>> I had my concerns regarding how sensitive is the subject of the
>>>>>>>>>>>> project ,but I think we should allow project leaders to develop their
>>>>>>>>>>>> communication strategy even if this has conflict of interest. It all
>>>>>>>>>>>> depends how they behave and how they manage this.
>>>>>>>>>>>> On Tuesday, October 6, 2015, Michael Coates <
>>>>>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>>>>>> It's not really that formal to add to the agenda, just a wiki
>>>>>>>>>>>>> that we add in the text.
>>>>>>>>>>>>> I think you can safely assume it will get the appropriate
>>>>>>>>>>>>> discussion.
>>>>>>>>>>>>> On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>>>>>>>>> Really?? Its not on the agenda yet for the next meeting??
>>>>>>>>>>>>> How does it get added to the agenda?
>>>>>>>>>>>>> And that was a formal request if that makes any difference :)
>>>>>>>>>>>>> I'm all in favour of getting the facts straight before any
>>>>>>>>>>>>> actions are taken, hence my request for an 'ethical review' or whatever it
>>>>>>>>>>>>> should be called.
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Simon
>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates <
>>>>>>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>> First step is to get all of our information straight so we're
>>>>>>>>>>>>>> clear on where things are at.
>>>>>>>>>>>>>> This was not on the board agenda last meeting and is also not
>>>>>>>>>>>>>> on the next agenda as of yet (of course it could always be added if
>>>>>>>>>>>>>> needed).
>>>>>>>>>>>>>> We are aware that people have raised questions though.   I'm
>>>>>>>>>>>>>> hoping we can get a clear understanding of all the facts and then discuss
>>>>>>>>>>>>>> if changes are needed.
>>>>>>>>>>>>>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Hey Michael,
>>>>>>>>>>>>>> Is the board going to take any action?
>>>>>>>>>>>>>> Were there any discussions about this controversy in the
>>>>>>>>>>>>>> board meeting at AppSec USA?
>>>>>>>>>>>>>> If not will it be on the agenda for the meeting on October
>>>>>>>>>>>>>> 14th?
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates <
>>>>>>>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>> I posted the below message earlier today. At this point my
>>>>>>>>>>>>>>> goal is to just gain clarity over the current reality and ideally drive to
>>>>>>>>>>>>>>> a shared state of success. This message doesn't seem to be reflected in the
>>>>>>>>>>>>>>> list yet. It could be because my membership hasn't been approved or because
>>>>>>>>>>>>>>> of mail list delays (I miss Google groups). But I think these questions
>>>>>>>>>>>>>>> will start the conversation.
>>>>>>>>>>>>>>> (This was just me asking questions as a curious Owasp
>>>>>>>>>>>>>>> member, not any action on behalf of the board)
>>>>>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>>>>> *From:* Michael Coates <michael.coates at owasp.org>
>>>>>>>>>>>>>>> *Date:* October 5, 2015 at 6:20:23 PM PDT
>>>>>>>>>>>>>>> *To:* owasp-benchmark-project at lists.owasp.org
>>>>>>>>>>>>>>> *Subject:* *Project Questions*
>>>>>>>>>>>>>>> OWASP Benchmark List,
>>>>>>>>>>>>>>> I've heard more about this project and am excited about the
>>>>>>>>>>>>>>> idea of an independent perspective of tool performance. I'm trying to
>>>>>>>>>>>>>>> understand a few things to better respond to questions from those in the
>>>>>>>>>>>>>>> security & OWASP community.
>>>>>>>>>>>>>>> In my mind there are two big areas for consideration in a
>>>>>>>>>>>>>>> benchmark process.
>>>>>>>>>>>>>>> 1. Are the benchmarks testing the right areas?
>>>>>>>>>>>>>>> 2. Is the process for creating the benchmark objective &
>>>>>>>>>>>>>>> free from conflicts of interest.
>>>>>>>>>>>>>>> I think as a group OWASP is the right body to align on #1.
>>>>>>>>>>>>>>> I'd like to ask for some clarifications on item #2. I think
>>>>>>>>>>>>>>> it's important to avoid actual conflict of interest and also the appearance
>>>>>>>>>>>>>>> of conflict of interest. The former is obvious why we mustn't have that,
>>>>>>>>>>>>>>> the latter is critical so others have faith in the tool, process and
>>>>>>>>>>>>>>> outputs of the process when viewing or hearing about the project.
>>>>>>>>>>>>>>> 1) Can we clarify whether other individuals have submitted
>>>>>>>>>>>>>>> meaningful code to the project?
>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>> Nearly all the code commits have come from 1 person (project
>>>>>>>>>>>>>>> lead).
>>>>>>>>>>>>>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>>>>>>>>>>>>>> 2) Can we clarify the contributions of others and their
>>>>>>>>>>>>>>> represented organizations?
>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>> The acknowledgements tab listed two developers (Juan Gama &
>>>>>>>>>>>>>>> Nick Sanidas) both who work at the same company as the project lead. It
>>>>>>>>>>>>>>> seems other people have submitted some small amounts of material, but
>>>>>>>>>>>>>>> overall it seems all development has come from the same company.
>>>>>>>>>>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>>>>>>>>>>>> 3) Can we clarify in what ways we've mitigated the potential
>>>>>>>>>>>>>>> conflict of interest and also the appearance of a conflict of interest?
>>>>>>>>>>>>>>> This seems like the largest blocker for wide spread acceptance of this
>>>>>>>>>>>>>>> project and the biggest risk.
>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>> The project lead and both of the project developers works
>>>>>>>>>>>>>>> for a company with very close ties to one of the companies that is
>>>>>>>>>>>>>>> evaluated by this project. Further, it appears the company is performing
>>>>>>>>>>>>>>> very well on the project tests.
>>>>>>>>>>>>>>> 4) If we are going to list tool vendors then I'd recommend
>>>>>>>>>>>>>>> listing multiple vendors for each category.
>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>> The tools page only lists 1 IAST tool. Since this is the
>>>>>>>>>>>>>>> point of the potential conflict of interest it is important to list
>>>>>>>>>>>>>>> numerous IAST tools.
>>>>>>>>>>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
>>>>>>>>>>>>>>> 5) Diverse body with multiple points of view
>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>> There is no indication that multiple stakeholders are
>>>>>>>>>>>>>>> present to review and decide on the future of this project. If they exist,
>>>>>>>>>>>>>>> a new section should be added to the project page to raise awareness. If
>>>>>>>>>>>>>>> they don't exist, we should reevaluate how we are obtaining an independent
>>>>>>>>>>>>>>> view of the testing process.
>>>>>>>>>>>>>>> Again, I think the idea of the project is great. From my
>>>>>>>>>>>>>>> perspective clarifying these questions will help ensure the project is not
>>>>>>>>>>>>>>> only objective, but also perceived as objective from someone reviewing the
>>>>>>>>>>>>>>> material. Ultimately this will contribute to the success and growth of the
>>>>>>>>>>>>>>> project.
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Michael Coates
>>>>>>>>>>>>>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> OK, based on the concerns raised so far I think the board
>>>>>>>>>>>>>>> should initiate a review of the OWASP Benchmark project.
>>>>>>>>>>>>>>> I'm not raising a formal complaint against it, I'm just
>>>>>>>>>>>>>>> requesting a review.
>>>>>>>>>>>>>>> And I dont think it needs a 'standard' project review -
>>>>>>>>>>>>>>> Johanna has already done a very good job of this.
>>>>>>>>>>>>>>> Not sure what sort of review you'd call it, I'll leave the
>>>>>>>>>>>>>>> naming to others :)
>>>>>>>>>>>>>>> I'm concerned that we have an OWASP project lead by a
>>>>>>>>>>>>>>> company who has a clear commercial stake in the results.
>>>>>>>>>>>>>>> Bringing more companies on board will help, but I'm still
>>>>>>>>>>>>>>> not sure that alone will make it independent enough.
>>>>>>>>>>>>>>> Commercial companies can afford to dedicate staff to
>>>>>>>>>>>>>>> improving Benchmark so that their products look better.
>>>>>>>>>>>>>>> Open source projects just cant do that, so we are at a
>>>>>>>>>>>>>>> distinct disadvantage.
>>>>>>>>>>>>>>> Should we allow a commercially driven OWASP project who's
>>>>>>>>>>>>>>> aim could be seen be to promote commercial software?
>>>>>>>>>>>>>>> If so, what sort of checks and balances does it need?
>>>>>>>>>>>>>>> Those are the sort of questions I'd like an independent
>>>>>>>>>>>>>>> review to look at.
>>>>>>>>>>>>>>> I do think there are some immediate steps that could be
>>>>>>>>>>>>>>> taken:
>>>>>>>>>>>>>>>    - I'd like to see the Benchmark project page clearly
>>>>>>>>>>>>>>>    state thats its at a very early stage and that the results are _not_ yet
>>>>>>>>>>>>>>>    suitable for use in commercial literature.
>>>>>>>>>>>>>>>    - I'd also like the main companies developing Benchmark
>>>>>>>>>>>>>>>    to be clearly stated on the main page. If and when other companies get
>>>>>>>>>>>>>>>    involved then this would actually help the project's claim of vendor
>>>>>>>>>>>>>>>    independence.
>>>>>>>>>>>>>>>    - And I'd love to see a respected co-leader added to the
>>>>>>>>>>>>>>>    project who is not associated with any commercial or open source security
>>>>>>>>>>>>>>>    tools:)
>>>>>>>>>>>>>>> And we should carry on discussing the project on this list -
>>>>>>>>>>>>>>> I think such discussions are very healthy, and I'd love to see this project
>>>>>>>>>>>>>>> mature to a state where it can be a trusted, independent and valued
>>>>>>>>>>>>>>> resource.
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias <
>>>>>>>>>>>>>>> tobias.gondrom at owasp.org> wrote:
>>>>>>>>>>>>>>>> @Simon:
>>>>>>>>>>>>>>>> yes, the leaders list is the place for your discussions for
>>>>>>>>>>>>>>>> project and chapter leaders
>>>>>>>>>>>>>>>> @Timo: I like your framing of "Don't ask what OWASP can do
>>>>>>>>>>>>>>>> for me, ask what I can do for OWASP."
>>>>>>>>>>>>>>>> That should and is indeed the spirit of OWASP:-)
>>>>>>>>>>>>>>>> Best regards, Tobias
>>>>>>>>>>>>>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>>>>>>>>>>>>>> I don't know enough about the matter to comment on this
>>>>>>>>>>>>>>>> case, but I feel that any situation where an OWASP project or any OWASP
>>>>>>>>>>>>>>>> initiative for that matter, is using OWASP to promote its own business
>>>>>>>>>>>>>>>> interests should be stopped.  We need to get rid of bad apples in OWASP.
>>>>>>>>>>>>>>>> OWASP is becoming a brand if you would like to think of it
>>>>>>>>>>>>>>>> that way and we are going to see many more cases of people trying to use
>>>>>>>>>>>>>>>> OWASP to spread
>>>>>>>>>>>>>>>> ...
> [Message clipped]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151127/e9e5aaf5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot 2015-11-27 16.59.28.png
Type: image/png
Size: 196334 bytes
Desc: not available
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151127/e9e5aaf5/attachment-0001.png>

More information about the OWASP-Leaders mailing list