[Owasp-board] [Owasp-leaders] OWASP Benchmark project - potential conflict of interest

Josh Sokol josh.sokol at owasp.org
Sat Nov 28 00:40:20 UTC 2015

One of the ideas that Andrew proposed was actually approaching WAFEC to
learn more about how they do vendor assessment in a neutral way.  It's
great to hear that we have a resource here already that we can leverage.  I
wasn't aware of your affiliation.

On Nov 27, 2015 2:47 PM, "Tony Turner" <tony.turner at owasp.org> wrote:

> I sincerely hope so. That's not the impression I got from others comments.
> Personally I haven't used the tool at all, but as I'm the project lead for
> another product evaluation project (WAFEC) I'm very sensitive to the need
> of collaboration with many different vendors. There really has to be a very
> high level (almost paranoid level) transparency with how vendors are
> approached, worked with, how requirements for evaluation are defined, and
> how metrics are derived.
> It appears the project team is attempting to address these last 2 somewhat
> but I'd like to see more specifics, and the lack of information on how they
> are addressing vendor communication, participation and transparency seems a
> bit concerning. Lastly, it is my opinion that project leadership should not
> belong to anyone working for or with a partnership/ownership stake for any
> vendor being evaluated. I think this is a flawed model and should
> transition to a vendor neutral party.
> On Nov 27, 2015 3:16 PM, "Josh Sokol" <josh.sokol at owasp.org> wrote:
>> I don't know what qualifies as "significant" in your mind, but my
>> understanding is that there have been contributions from other vendors:
>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>> Still, Dave would like more, but he can't force them to help.
>> ~josh
>> On Fri, Nov 27, 2015 at 1:45 PM, Tony Turner <tony.turner at owasp.org>
>> wrote:
>>> While I can appreciate that they started with Contrast, if there hasn't
>>> been significant effort to include other vendors it's a worthless
>>> benchmark. It's easy to state you haven't gotten support from other vendors
>>> and that's fine, but until you do there's really nothing to release. Why
>>> was it ever upgraded? Talking about the results without an accurate
>>> comparative analysis is akin to snake oil.
>>> On Nov 27, 2015 1:49 PM, "Josh Sokol" <josh.sokol at owasp.org> wrote:
>>>> Thank you for the links to those articles.  The first one discusses the
>>>> strengths and weaknesses of the different methods of evaluating for
>>>> application vulnerabilities.  The section on the Benchmark seems wholly
>>>> appropriate to me.  That seems like an excellent description of what the
>>>> project is designed to do.  I see some metrics in there about which tools
>>>> are more effective on which types of vulnerabilities, but I don't see him
>>>> straight up saying "The OWASP Benchmark proves that Contrast is better".
>>>> This seems like statements made based on some level of testing and
>>>> research.  Honestly, I don't see any OWASP brand abuse in that article.
>>>> Whether it's in good taste or not at this stage in the project is certainly
>>>> debatable, but if you look at the brand usage guidelines (
>>>> https://www.owasp.org/index.php/Marketing/Resources#tab=BRAND_GUIDELINES),
>>>> I don't see any violations.  We need to govern to policy here which is why
>>>> Paul and Noreen are evaluating changes to the guidelines and our
>>>> enforcement policies to make abuse more difficult.
>>>> The second article is a competing vendor's reaction to the first.  He
>>>> makes some good points about the issues with Benchmark, but he also says
>>>> that he hopes that it will be improved over time, and Dave has committed to
>>>> that.  What I don't see is the vendor saying "...and Veracode has committed
>>>> resources to help make the Benchmark more accurate across all tool sets".
>>>> The Benchmark page is pretty clear that it does it's best to provide a
>>>> benchmark without working exactly like a real-world application.  Maybe
>>>> some more disclaimer text about where the project is at today would be in
>>>> order to validate some of Chris' concerns, but I hardly see this as "brand
>>>> abuse" or a reason to demote the project.
>>>> Please consider that I have spoken with both Dave and Jeff on this
>>>> topic and read much of the discussions around it before formulating my
>>>> opinion.  I doubt that you have done the same so I'm not sure how you can
>>>> claim that you have researched the issues and all parties involved when you
>>>> haven't even spoken with the two people whom you are accusing of
>>>> impropriety.  I have no bias here.  I am simply speaking with the
>>>> individuals involved, looking at the currently OWASP policies and
>>>> guidelines, and helping to determine our next steps.
>>>> ~josh
>>>> On Fri, Nov 27, 2015 at 12:22 PM, johanna curiel curiel <
>>>> johanna.curiel at owasp.org> wrote:
>>>>> >>While I agree with you that there has been some brand abuse, it was
>>>>> abuse by Contrast (specifically their marketing department), and not by
>>>>> "these gentlemen" as  you state.
>>>>> Really? ..'some brand abuse'..this is more than brand abuse
>>>>> Josh , please read also the article written by Jeff
>>>>> http://www.darkreading.com/vulnerabilities---threats/why-its-insane-to-trust-static-analysis/a/d-id/1322274
>>>>> ?
>>>>> And Veracode's reaction including others in Twitter
>>>>> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>>>> My strong advice is to research the issues and all the parties
>>>>> involved before making statements
>>>>> On Fri, Nov 27, 2015 at 2:07 PM, Josh Sokol <josh.sokol at owasp.org>
>>>>> wrote:
>>>>>> Jim,
>>>>>> A concern was expressed to the Board and, frankly, I am insulted by
>>>>>> you saying that this was "brushed under the rug".  The Board delegated Matt
>>>>>> to talk with Dave and they had a lengthy conversation on the subject.  The
>>>>>> Board delegated me to talk with Jeff and we had a lengthy conversation on
>>>>>> the subject.  If you do not trust in our abilities to read people, ask the
>>>>>> right questions, and provide honest feedback about our conversations, then
>>>>>> that's a bigger issue that we should take offline.  After our
>>>>>> conversations, we took the time to call a special two-hour session of the
>>>>>> Board in order to discuss this subject (and only this subject).  We spoke
>>>>>> about all facets of the issue at hand, about the challenges and possible
>>>>>> solutions, and concluded on some very concrete next steps.
>>>>>> While I agree with you that there has been some brand abuse, it was
>>>>>> abuse by Contrast (specifically their marketing department), and not by
>>>>>> "these gentlemen" as  you state.  Unless you can point to some sort of
>>>>>> evidence showing that Jeff and/or Dave first-hand abused the brand, then I
>>>>>> believe that you are speaking with your heart instead of with your head.  I
>>>>>> appreciate your passion, but I label this as conspiracy theory because
>>>>>> without evidence to support your claims, I cannot accept it as anything
>>>>>> other.
>>>>>> ~josh
>>>>>> On Fri, Nov 27, 2015 at 11:39 AM, Jim Manico <jim.manico at owasp.org>
>>>>>> wrote:
>>>>>>> Josh,
>>>>>>> I stand by my comments and perspective, but I'm disheartened that
>>>>>>> you consider my presentation of facts (and the concerns of many active
>>>>>>> members of our community) as a "conspiracy theory".
>>>>>>> In my experience, these kind of comments border on insults and only
>>>>>>> cause folks to harden their opinions.
>>>>>>> Once again I feel these gentlemen got away with a kind of brand
>>>>>>> abuse that is very hurtful to the OWASP community but I am at a loss as to
>>>>>>> how handle or prevent these kinds of mishaps - especially when board
>>>>>>> members like yourself seem willing to - from what I see - brush it under
>>>>>>> the rug.
>>>>>>> --
>>>>>>> Jim Manico
>>>>>>> Global Board Member
>>>>>>> OWASP Foundation
>>>>>>> https://www.owasp.org
>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>> On Nov 27, 2015, at 7:23 PM, Josh Sokol <josh.sokol at owasp.org>
>>>>>>> wrote:
>>>>>>> Admittedly, this was my gut reaction at first as well.  I began
>>>>>>> linking all of these companies, people, and projects together in my mind
>>>>>>> (there are some loose links there) and painted a big conspiracy picture
>>>>>>> similar to what Jim and Dinis have stated.  But, after speaking directly
>>>>>>> with Jeff, and hearing about the conversation that Dave and Matt had, I've
>>>>>>> changed my mind.
>>>>>>> I think it begins with the project itself.  If you aren't sold on
>>>>>>> the idea of the Benchmark, then you'll never be able to get to the same
>>>>>>> place.  My original line of thinking was that it was just a bar for vendors
>>>>>>> to compare their tools against eachother, but that's a bit myopic.  We are
>>>>>>> in an industry where things evolve very quickly.  As a customer of these
>>>>>>> tools, I know firsthand that something that a tool does today may not be
>>>>>>> the case a week from now.  Likewise, new features are being added daily and
>>>>>>> I need a point-in-time metric to be able to gauge continual effectiveness.
>>>>>>> Cool, right?  But not a game changer.  The game changer part comes when you
>>>>>>> realize that by developing and evolving the tests that go into the
>>>>>>> Benchmark, we are moving the bar higher and higher.  We (OWASP) are
>>>>>>> effectively setting the standard by which these tools will be compared.  A
>>>>>>> tool that receives a lower score on the Benchmark today knows exactly what
>>>>>>> they need to work on in order to pass that test tomorrow and we already
>>>>>>> have examples of tools that have made improvements because of their
>>>>>>> Benchmark score (Ask Simon about ZAP's experience with the Benchmark).  I
>>>>>>> don't think that anyone can argue that the Benchmark project isn't being
>>>>>>> effective when OWASP's own tools are being driven forward as a result of
>>>>>>> using it.
>>>>>>> But, but, but, Dave and Jeff own Aspect and have stock in Contrast
>>>>>>> and Jeff is the Contrast CTO and Contrast got good scores so it's a
>>>>>>> conspiracy right?  Is there some code that allows Contrast to use the
>>>>>>> Benchmark?  Absolutely.  Can you really blame Dave for starting his testing
>>>>>>> on the effectiveness of the Benchmark with a tool that he owned and is
>>>>>>> familiar with?  If I were going to start a similar project, there's no
>>>>>>> question in my mind that I would begin my testing with the tools that I
>>>>>>> have available to me.  That said, is there code that allows other tools to
>>>>>>> use the Benchmark?  Absolutely.
>>>>>>> Regarding "Dave has a history of breaching his duty to be vendor
>>>>>>> neutral", while I cannot comment on his past actions, I can judge what
>>>>>>> we've seen recently.  Matt saw a presentation from Dave on the Benchmark at
>>>>>>> a conference in Chicago.  He said that he felt that the message was
>>>>>>> appropriate and while IAST tools were mentioned as receiving higher scores,
>>>>>>> it wasn't a "Contrast is the best" type of message, more of a generality.
>>>>>>> I saw a very similar (if not the same) talk by Jeff at LASCON 2015 and the
>>>>>>> message was exactly the same.  I watched the talk expecting some sort of
>>>>>>> impropriety, but found none.  So, perhaps Dave has abused some privilege
>>>>>>> granted to him in the past, but what I've seen from him at this point, with
>>>>>>> respect to the Benchmark, has been appropriate.
>>>>>>> You have a very good point with respect to the Contrast marketing
>>>>>>> message around the Benchmark.  It's been completely absurd, over the top,
>>>>>>> and, in my personal opinion, intolerable.  In fact, I experienced the same
>>>>>>> thing that you talked about with them at LASCON 2015 where they stood in
>>>>>>> front of the door of the room Jeff was speaking in and scanned attendees as
>>>>>>> they went into the talk.  I agree that these types of aggressive marketing
>>>>>>> tactics cannot be tolerated at OWASP.  In addition, we have seen several
>>>>>>> marketing messages from them effectively implying that OWASP endorses
>>>>>>> Contrast.  Clearly this is not OK.  I've spoken with Jeff about it and we
>>>>>>> agreed that it is not in the Benchmark's best interest to have this
>>>>>>> aggressive Contrast marketing around it at such an early stage.  He has
>>>>>>> said that he is not responsible for Contrast's marketing team, but that he
>>>>>>> would speak with the people who are.  I haven't seen a single message from
>>>>>>> them since so I'm guessing that he's made good on this promise.  While
>>>>>>> that's an excellent start, OWASP's takeaway here should be that we need to
>>>>>>> do a better job with our brand usage guidelines both in terms of the
>>>>>>> wording and enforcement.  There are many other companies out there that use
>>>>>>> the OWASP brand and I think that we agree that selective enforcement
>>>>>>> against Contrast is not the right answer.  Paul and Noreen are actively
>>>>>>> working on this.  Either way, I think that implying that activities from a
>>>>>>> vendor's marketing department means that the project is not objective is
>>>>>>> not inappropriate.  If we feel that the project is not objective, then
>>>>>>> separate measures need to be taken to drive contribution diversity into
>>>>>>> it.  That I absolutely agree with and the message from Dave was that he
>>>>>>> would love to have more contributors to his project.  But, seeing as we
>>>>>>> cannot force people to work on it, this becomes a matter of "put up or shut
>>>>>>> up".  The same goes for the experts that you said reviewed the code.  If
>>>>>>> they feel that it is somehow skewed towards Contrast, they have the power
>>>>>>> to change that.  Now, if someone tries to participate and Dave tells them
>>>>>>> "No thanks", then I agree we have a problem, but I don't hear anyone
>>>>>>> inferring that happened.
>>>>>>> Please, let's drop the conspiracy theories and focus on the tangible
>>>>>>> things that we can do to help an OWASP project to be more successful.  Help
>>>>>>> find more participants to drive diversity, update our brand usage
>>>>>>> guidelines to prevent abuse, enforce them widely, etc.  Thank you.
>>>>>>> ~josh
>>>>>>> On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico <jim.manico at owasp.org>
>>>>>>> wrote:
>>>>>>>> Dinis,
>>>>>>>> Like a rare celestial moment when all the planets plus Pluto are
>>>>>>>> aligned, I just read your email on the future of OWASP projects thinking,
>>>>>>>> "Dinis is spot on".
>>>>>>>> Reflecting on projects I manage or work on...
>>>>>>>> The Java Encoder and HTML Sanitizer are likely best moved to Apache
>>>>>>>> now that they have reached a measure of adoption and maturity. Apache would
>>>>>>>> be a much better long term custodian. Perhaps the same for AppSensor, but
>>>>>>>> not my project - just thinking out loud.
>>>>>>>> Other similar defensive projects are still being noodled on, so
>>>>>>>> OWASP is a decent home for these research efforts.
>>>>>>>> The whole tools category is also something to consider. Dependency
>>>>>>>> Check and of course ZAP are some of the best projects that OWASP offers,
>>>>>>>> are they best served where they are today? Both have rich communities of
>>>>>>>> developers but I don't see the foundation doing much to support these
>>>>>>>> efforts.
>>>>>>>> ASVS has the opportunity to effect massive change, I would to love
>>>>>>>> to see major investment and volunteer activity here. Pro tech writer,
>>>>>>>> detailed discourses on each individual requirement, etc. If I was king (and
>>>>>>>> I am not, at all) I would invest in ASVS on a 6 figure scale. (And who
>>>>>>>> started ASVS? Jeff, Dave and Boberski, hat tip to such a marvelous idea).
>>>>>>>> Or maybe moving ASVS to the W3C or IETF would help it grow?
>>>>>>>> The Proactive Controls was a pet project but as we approach 2.0 we
>>>>>>>> have several active/awesome volunteers working on it. We will be making the
>>>>>>>> doc "world editable" to make contributions easy. OWASP seems like a good
>>>>>>>> home for such an awareness doc. Same with T10, especially if community
>>>>>>>> edits are welcome.
>>>>>>>> Anyhow, I'm with you on this Dinis. Once a project starts to reach
>>>>>>>> production quality, spinning off the project as an external project or
>>>>>>>> moving it to a different foundation where managing production software or
>>>>>>>> formal standards is their thing seems realistic.
>>>>>>>> I don't have all the answers here, but your email certainly
>>>>>>>> resonated with me.
>>>>>>>> Aloha,
>>>>>>>> --
>>>>>>>> Jim Manico
>>>>>>>> Global Board Member
>>>>>>>> OWASP Foundation
>>>>>>>> https://www.owasp.org
>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>> On Nov 26, 2015, at 11:26 PM, Dinis Cruz <dinis.cruz at owasp.org>
>>>>>>>> wrote:
>>>>>>>> Jim's reading of this situation is exactly my view on the value of
>>>>>>>> the Contrast tool and how it has been 'pushing' the rules of engagement to
>>>>>>>> an very 'fuzzy' moral/ethical/commercial limit :)
>>>>>>>> As per my last email, a key problem here is the 'perceived
>>>>>>>> expectation' of what is an OWASP project, and how it should be consumed.
>>>>>>>> If you look at the OWASP benchmark as a research project, then the
>>>>>>>> only way it could be making the kind of claims it makes (and have
>>>>>>>> credibility) is if it had evolved from OWASP, with its own (diverse)
>>>>>>>> community
>>>>>>>> On 26 November 2015 at 21:01, Jim Manico <jim.manico at owasp.org>
>>>>>>>> wrote:
>>>>>>>>> I have a different take on this situation but my opinion is the
>>>>>>>>> "minority opinion". I will respect the rest of the boards take on this, but
>>>>>>>>> here is how I see it.
>>>>>>>>> First of all, Jeff has stated that he feels I am attacking him
>>>>>>>>> personally from a past personal grudge, and frankly I do not fault him for
>>>>>>>>> that perspective since we definitely have history with conflict. So it's
>>>>>>>>> fair to take my opinion on this with a grain of salt.
>>>>>>>>> I look at this situation from the perspective of a forensic
>>>>>>>>> investigator.
>>>>>>>>> 1) The Benchmark project had Contrast hooks and only Contrast
>>>>>>>>> hooks in it when I reviewed it so this leads me to believe that the project
>>>>>>>>> was clearly built with Contrast in mind from the ground up, at least in
>>>>>>>>> some way.
>>>>>>>>> 3) Dave has a history of breaching his duty to be vendor neutral.
>>>>>>>>> He was gifted with a keynote in South Korea a few years ago, and used that
>>>>>>>>> opportunity to discuss and pitch Contrast, on stage, during a keynote -
>>>>>>>>> with Contrast specific slides. This is just supporting evidence of his
>>>>>>>>> intention at OWASP to push Contrast in ways that I think are against the
>>>>>>>>> intentions and goals of our foundation.
>>>>>>>>> 3) Other experts have reviewed the project and felt that many of
>>>>>>>>> the tests were very slanted and almost contrived to support Contrast. I can
>>>>>>>>> drag those folks into this conversation, but I do not think that would help
>>>>>>>>> in any way. So it's fair to call this point heresy.
>>>>>>>>> 4) I do not see this project as revolutionary, at all. Every
>>>>>>>>> vendor has their own test suite tuned for their tool. As the benchmark
>>>>>>>>> stands today, I see it as just another vendors product-specific benchmark.
>>>>>>>>> Mass collaboration from many vendors is not just a "nice to have" but a
>>>>>>>>> base requirement to get even close to useful for objective tool measurement.
>>>>>>>>> 5) Jeff stating that his Marketing people went over the line is
>>>>>>>>> also an admission that - well, they went over the line. By the same token
>>>>>>>>> Jeff was in his booth at AppSec USA surrounded by benchmark marketing
>>>>>>>>> material, discussing this to prospects and he even asked me and Mr Coates
>>>>>>>>> to wade into this debate and support Dave. So to say he was not involved
>>>>>>>>> and it was only his marketing people seems a stretch at best.
>>>>>>>>> 6) The Contrast marketing team was wandering around the conference
>>>>>>>>> zapping folks to get leads, and I asked them to stay in their booth, which
>>>>>>>>> is standard conference policy. These folks know better but are again going
>>>>>>>>> over the line to sell product at OWASP. There is a better way (like
>>>>>>>>> focusing on product capability and language support, have consistent +
>>>>>>>>> stellar customer service, have a humble and gracious attitude to all
>>>>>>>>> prospects and customers, actively participate in OWASP in a vendor neutral
>>>>>>>>> and community supportive way, etc).
>>>>>>>>> Please note, I think Contrast is a decent tool, I've offered to
>>>>>>>>> resell in the past, and I have recommended it in certain situations - even
>>>>>>>>> after this situation arose. I'm stating this out of honestly and desire to
>>>>>>>>> put my cards on the table. I truly want Jeff and Dave to be successful.
>>>>>>>>> They have dedicated their lives to AppSec and if anyone should win
>>>>>>>>> big-time, I hope it's them. I even told Jeff I hope he hits the mother load
>>>>>>>>> and donates a little back to OWASP.
>>>>>>>>> However, my instinct and evidence tell me that they both went over
>>>>>>>>> the line in the use of the OWASP brand to sell product.
>>>>>>>>> Now, Jeff makes a good point. We as a board and staff are very
>>>>>>>>> poor at enforcing brand management policy and it's not fair to single out
>>>>>>>>> Contrast, when many other vendors violate the brand, IMO. Just google OWASP
>>>>>>>>> and watch the ads fly that use the OWASP name to sell product.
>>>>>>>>> Also, any and every request that was made of Dave to adjust the
>>>>>>>>> project for the sake of vendor neutrality was taken very seriously.
>>>>>>>>> Regardless of Daves past intentions, he is clearly trying to do the right
>>>>>>>>> thing moving forward.
>>>>>>>>> I look to "postels principle" in this situation (this is otherwise
>>>>>>>>> known as the "robustness principle" and dates back to the creation of TCP)
>>>>>>>>> . This is paraphrased as, "Be liberal in what you take from others but be
>>>>>>>>> conservative in what you dish out". So I think it's critical that OWASP and
>>>>>>>>> any OWASP resource present itself in a strict vendor neutral way. But
>>>>>>>>> unless OWASP wants to be much more "even" in the enforcement of brand
>>>>>>>>> policy across the board to all violators, we should be fairly lax in the
>>>>>>>>> enforcement of these issues from the outside world.
>>>>>>>>> I am trying to be objective here. My trigonometry teacher once
>>>>>>>>> told me "I'd fail my mother" when I asked him if he would ever fail me (I
>>>>>>>>> was an A student). If my mother owned a security company and tried the same
>>>>>>>>> stunt, I'd have the same opinions about her actions as well.
>>>>>>>>> So what next? Well hello from the other side. I'm going back to
>>>>>>>>> listening to Adele's new album where I can sit in my deep feelings and
>>>>>>>>> reflect upon what the OWASP foundation has done to enrich my life. I would
>>>>>>>>> much rather keep out of this (and any other conflict laden situation at
>>>>>>>>> OWASP), but I feel it's my responsibility to speak up.
>>>>>>>>> Aloha,
>>>>>>>>> --
>>>>>>>>> Jim Manico
>>>>>>>>> Global Board Member
>>>>>>>>> OWASP Foundation
>>>>>>>>> https://www.owasp.org
>>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>> On Nov 26, 2015, at 9:09 PM, Josh Sokol <josh.sokol at owasp.org>
>>>>>>>>> wrote:
>>>>>>>>> I would be happy to provide an update.
>>>>>>>>>    - Matt Konda and Dave Wichers, the Benchmark Project Leader,
>>>>>>>>>    had a conversation a few weeks back.  To summarize their conversation, Dave
>>>>>>>>>    acknowledges the currently lack of diversity in his project and it is his
>>>>>>>>>    sincere desire to drive more people to it to help.  He also acknowledges
>>>>>>>>>    the issues with Contrast's extreme marketing around the project and feels
>>>>>>>>>    that it is in everyone's best interests for them to curb it back.  While he
>>>>>>>>>    does have an ownership stake in Contrast, he works at Aspect and has no
>>>>>>>>>    control over the marketing messages that they are putting out there.  From
>>>>>>>>>    the Board perspective, there has been no evidence of any impropriety on
>>>>>>>>>    Dave's part and it should be our goal to drive more diversity into the
>>>>>>>>>    project to support Dave.  Dave appears to be sincere in his desires to
>>>>>>>>>    create a tool where OWASP can tell vendors what we expect from their
>>>>>>>>>    tools.  If the main issue is that only members of Aspect are working on it,
>>>>>>>>>    then the best thing that we can do is try to get him some outside
>>>>>>>>>    assistance.  We are also asking that the project be opened up to commits
>>>>>>>>>    via Git so that outsiders can push commits to it.
>>>>>>>>>    - Josh Sokol and Jeff Williams, the CTO of Contrast, had a
>>>>>>>>>    conversation a few weeks back.  To summarize their conversation, Jeff
>>>>>>>>>    believes that the work that Dave is doing on the Benchmark is a game
>>>>>>>>>    changer in that it gives OWASP the power in dictating what these tools need
>>>>>>>>>    to be finding.  He wants the Benchmark to be successful and understands
>>>>>>>>>    that it needs to be diverse in order to be trusted.  He recognizes that
>>>>>>>>>    Dave is trying to do that and does not want the marketing message from
>>>>>>>>>    Contrast to interfere with his efforts.  Jeff felt that the "Lab" status
>>>>>>>>>    granted to Benchmark meant that it was ready for mainstream adoption, that
>>>>>>>>>    it had 21k tests, and was almost a year old, and didn't see anything wrong
>>>>>>>>>    with marketing their results, but has agreed to talk to their marketing
>>>>>>>>>    team to get them to lay off that message for now.  From the Board
>>>>>>>>>    perspective, we have come to the realization that our brand usage
>>>>>>>>>    guidelines need an overhaul to clarify what is and is not allowed.  We have
>>>>>>>>>    made a few proposals and have reached out to Mozilla to gain more insight
>>>>>>>>>    on their guidelines and even ask for assistance.  Noreen and Paul are
>>>>>>>>>    taking lead on these efforts.
>>>>>>>>>    - There is a note in the notes that the Board was supposed to
>>>>>>>>>    follow up with an open letter to the community and companies involved
>>>>>>>>>    describing our review and actions.  I don't think that has happened so I
>>>>>>>>>    will remind the person who took on that action item.
>>>>>>>>> I'm happy to answer any questions that you may have.
>>>>>>>>> ~josh
>>>>>>>>> On Thu, Nov 26, 2015 at 11:55 AM, Tobias <tobias.gondrom at owasp.org
>>>>>>>>> > wrote:
>>>>>>>>>> There have been several conversations on that matter and a
>>>>>>>>>> dedicated call. Unfortunately for personal reasons I could not attend the
>>>>>>>>>> last call as it was at 04:00am my local time, but all other board members
>>>>>>>>>> did participate.
>>>>>>>>>> Could please one of my fellow board members give an update.
>>>>>>>>>> Best, Tobias
>>>>>>>>>> On 26/11/15 18:04, Timo Goosen wrote:
>>>>>>>>>> I would also like to know the answer to Simon's question. We need
>>>>>>>>>> to get rid of bad apples in OWASP in my opinion, there are too many people
>>>>>>>>>> just using the OWASP "name" or "brand" to improve their own financial
>>>>>>>>>> situation or career.
>>>>>>>>>> Regards.
>>>>>>>>>> Timo
>>>>>>>>>> On Thu, Nov 26, 2015 at 1:13 PM, psiinon <psiinon at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> Paul, and the rest of the board,
>>>>>>>>>>> Its been over 2 months since I raised this issue.
>>>>>>>>>>> Whats happening?
>>>>>>>>>>> Has the board even discussed it?
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Simon
>>>>>>>>>>> On Tue, Oct 20, 2015 at 10:00 PM, Paul Ritchie <
>>>>>>>>>>> paul.ritchie at owasp.org> wrote:
>>>>>>>>>>>> Eoin, Johanna, All:
>>>>>>>>>>>> In an earlier email, Josh Sokol mentioned that he will be
>>>>>>>>>>>> speaking in the next day or 2 to their CTO, while at LASCON, as a
>>>>>>>>>>>> representative of the OWASP Board.  Following that feedback, the Board has
>>>>>>>>>>>> action to take the next steps.
>>>>>>>>>>>> Just an FYI that all comments are recognized and action is
>>>>>>>>>>>> being taken.
>>>>>>>>>>>> Paul
>>>>>>>>>>>> Best Regards, Paul Ritchie
>>>>>>>>>>>> OWASP Executive Director
>>>>>>>>>>>> paul.ritchie at owasp.org
>>>>>>>>>>>> On Tue, Oct 20, 2015 at 1:54 PM, johanna curiel curiel <
>>>>>>>>>>>> johanna.curiel at owasp.org> wrote:
>>>>>>>>>>>>> Time for owasp to do a public statement and put a clear story
>>>>>>>>>>>>> regarding this abusive behavior of Owasp brand
>>>>>>>>>>>>> On Tuesday, October 20, 2015, Eoin Keary <eoin.keary at owasp.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Folks,
>>>>>>>>>>>>>> The project should be immediately shelved it's simply bad
>>>>>>>>>>>>>> form.
>>>>>>>>>>>>>> This is damaging to OWASP, the industry and exactly what
>>>>>>>>>>>>>> OWASP is not about.
>>>>>>>>>>>>>> There is a clear conflict of interest and distinct lack of
>>>>>>>>>>>>>> science behind the claims made by Contrast.
>>>>>>>>>>>>>> Eoin Keary
>>>>>>>>>>>>>> OWASP Volunteer
>>>>>>>>>>>>>> @eoinkeary
>>>>>>>>>>>>>> On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel <
>>>>>>>>>>>>>> johanna.curiel at owasp.org> wrote:
>>>>>>>>>>>>>> At the moment we did the project review, we observed that the
>>>>>>>>>>>>>> project did not have enough testing to be considered in any form as 'ready'
>>>>>>>>>>>>>>  for benchmarking, neither that it had yet the community adoption, however
>>>>>>>>>>>>>> technically speaking as it has been classified by the leaders, the project
>>>>>>>>>>>>>> is at the beta stage.
>>>>>>>>>>>>>> Indeed , Dave had the push to have the project reviewed but
>>>>>>>>>>>>>> it was never clear that later on the project was going to be advertisied
>>>>>>>>>>>>>> this way. That all happend after the presentation at Appsec.
>>>>>>>>>>>>>> I had my concerns regarding how sensitive is the subject
>>>>>>>>>>>>>> of the project ,but I think we should allow project leaders to develop
>>>>>>>>>>>>>> their communication strategy even if this has conflict of interest. It all
>>>>>>>>>>>>>> depends how they behave and how they manage this.
>>>>>>>>>>>>>> On Tuesday, October 6, 2015, Michael Coates <
>>>>>>>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>> It's not really that formal to add to the agenda, just a
>>>>>>>>>>>>>>> wiki that we add in the text.
>>>>>>>>>>>>>>> I think you can safely assume it will get the appropriate
>>>>>>>>>>>>>>> discussion.
>>>>>>>>>>>>>>> On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Really?? Its not on the agenda yet for the next meeting??
>>>>>>>>>>>>>>> How does it get added to the agenda?
>>>>>>>>>>>>>>> And that was a formal request if that makes any difference :)
>>>>>>>>>>>>>>> I'm all in favour of getting the facts straight before any
>>>>>>>>>>>>>>> actions are taken, hence my request for an 'ethical review' or whatever it
>>>>>>>>>>>>>>> should be called.
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates <
>>>>>>>>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>>> First step is to get all of our information straight so
>>>>>>>>>>>>>>>> we're clear on where things are at.
>>>>>>>>>>>>>>>> This was not on the board agenda last meeting and is also
>>>>>>>>>>>>>>>> not on the next agenda as of yet (of course it could always be added if
>>>>>>>>>>>>>>>> needed).
>>>>>>>>>>>>>>>> We are aware that people have raised questions though.
>>>>>>>>>>>>>>>> I'm hoping we can get a clear understanding of all the facts and then
>>>>>>>>>>>>>>>> discuss if changes are needed.
>>>>>>>>>>>>>>>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> Hey Michael,
>>>>>>>>>>>>>>>> Is the board going to take any action?
>>>>>>>>>>>>>>>> Were there any discussions about this controversy in the
>>>>>>>>>>>>>>>> board meeting at AppSec USA?
>>>>>>>>>>>>>>>> If not will it be on the agenda for the meeting on October
>>>>>>>>>>>>>>>> 14th?
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates <
>>>>>>>>>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>> I posted the below message earlier today. At this point my
>>>>>>>>>>>>>>>>> goal is to just gain clarity over the current reality and ideally drive to
>>>>>>>>>>>>>>>>> a shared state of success. This message doesn't seem to be reflected in the
>>>>>>>>>>>>>>>>> list yet. It could be because my membership hasn't been approved or because
>>>>>>>>>>>>>>>>> of mail list delays (I miss Google groups). But I think these questions
>>>>>>>>>>>>>>>>> will start the conversation.
>>>>>>>>>>>>>>>>> (This was just me asking questions as a curious Owasp
>>>>>>>>>>>>>>>>> member, not any action on behalf of the board)
>>>>>>>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>>>>>>> *From:* Michael Coates <michael.coates at owasp.org>
>>>>>>>>>>>>>>>>> *Date:* October 5, 2015 at 6:20:23 PM PDT
>>>>>>>>>>>>>>>>> *To:* owasp-benchmark-project at lists.owasp.org
>>>>>>>>>>>>>>>>> *Subject:* *Project Questions*
>>>>>>>>>>>>>>>>> OWASP Benchmark List,
>>>>>>>>>>>>>>>>> I've heard more about this project and am excited about
>>>>>>>>>>>>>>>>> the idea of an independent perspective of tool performance. I'm trying to
>>>>>>>>>>>>>>>>> understand a few things to better respond to questions from those in the
>>>>>>>>>>>>>>>>> security & OWASP community.
>>>>>>>>>>>>>>>>> In my mind there are two big areas for consideration in a
>>>>>>>>>>>>>>>>> benchmark process.
>>>>>>>>>>>>>>>>> 1. Are the benchmarks testing the right areas?
>>>>>>>>>>>>>>>>> 2. Is the process for creating the benchmark objective &
>>>>>>>>>>>>>>>>> free from conflicts of interest.
>>>>>>>>>>>>>>>>> I think as a group OWASP is the right body to align on #1.
>>>>>>>>>>>>>>>>> I'd like to ask for some clarifications on item #2. I
>>>>>>>>>>>>>>>>> think it's important to avoid actual conflict of interest and also the
>>>>>>>>>>>>>>>>> appearance of conflict of interest. The former is obvious why we mustn't
>>>>>>>>>>>>>>>>> have that, the latter is critical so others have faith in the tool, process
>>>>>>>>>>>>>>>>> and outputs of the process when viewing or hearing about the project.
>>>>>>>>>>>>>>>>> 1) Can we clarify whether other individuals have submitted
>>>>>>>>>>>>>>>>> meaningful code to the project?
>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>> Nearly all the code commits have come from 1 person
>>>>>>>>>>>>>>>>> (project lead).
>>>>>>>>>>>>>>>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>>>>>>>>>>>>>>>> 2) Can we clarify the contributions of others and their
>>>>>>>>>>>>>>>>> represented organizations?
>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>> The acknowledgements tab listed two developers (Juan Gama
>>>>>>>>>>>>>>>>> & Nick Sanidas) both who work at the same company as the project lead. It
>>>>>>>>>>>>>>>>> seems other people have submitted some small amounts of material, but
>>>>>>>>>>>>>>>>> overall it seems all development has come from the same company.
>>>>>>>>>>>>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>>>>>>>>>>>>>> 3) Can we clarify in what ways we've mitigated the
>>>>>>>>>>>>>>>>> potential conflict of interest and also the appearance of a conflict of
>>>>>>>>>>>>>>>>> interest? This seems like the largest blocker for wide spread acceptance of
>>>>>>>>>>>>>>>>> this project and the biggest risk.
>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>> The project lead and both of the project developers works
>>>>>>>>>>>>>>>>> for a company with very close ties to one of the companies that is
>>>>>>>>>>>>>>>>> evaluated by this project. Further, it appears the company is performing
>>>>>>>>>>>>>>>>> very well on the project tests.
>>>>>>>>>>>>>>>>> 4) If we are going to list tool vendors then I'd recommend
>>>>>>>>>>>>>>>>> listing multiple vendors for each category.
>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>> The tools page only lists 1 IAST tool. Since this is the
>>>>>>>>>>>>>>>>> point of the potential conflict of interest it is important to list
>>>>>>>>>>>>>>>>> numerous IAST tools.
>>>>>>>>>>>>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
>>>>>>>>>>>>>>>>> 5) Diverse body with multiple points of view
>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>> There is no indication that multiple stakeholders are
>>>>>>>>>>>>>>>>> present to review and decide on the future of this project. If they exist,
>>>>>>>>>>>>>>>>> a new section should be added to the project page to raise awareness. If
>>>>>>>>>>>>>>>>> they don't exist, we should reevaluate how we are obtaining an independent
>>>>>>>>>>>>>>>>> view of the testing process.
>>>>>>>>>>>>>>>>> Again, I think the idea of the project is great. From my
>>>>>>>>>>>>>>>>> perspective clarifying these questions will help ensure the project is not
>>>>>>>>>>>>>>>>> only objective, but also perceived as objective from someone reviewing the
>>>>>>>>>>>>>>>>> material. Ultimately this will contribute to the success and growth of the
>>>>>>>>>>>>>>>>> project.
>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Michael Coates
>>>>>>>>>>>>>>>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> OK, based on the concerns raised so far I think the board
>>>>>>>>>>>>>>>>> should initiate a review of the OWASP Benchmark project.
>>>>>>>>>>>>>>>>> I'm not raising a formal complaint against it, I'm just
>>>>>>>>>>>>>>>>> requesting a review.
>>>>>>>>>>>>>>>>> And I dont think it needs a 'standard' project review -
>>>>>>>>>>>>>>>>> Johanna has already done a very good job of this.
>>>>>>>>>>>>>>>>> Not sure what sort of review you'd call it, I'll leave the
>>>>>>>>>>>>>>>>> naming to others :)
>>>>>>>>>>>>>>>>> I'm concerned that we have an OWASP project lead by a
>>>>>>>>>>>>>>>>> company who has a clear commercial stake in the results.
>>>>>>>>>>>>>>>>> Bringing more companies on board will help, but I'm still
>>>>>>>>>>>>>>>>> not sure that alone will make it independent enough.
>>>>>>>>>>>>>>>>> Commercial companies can afford to dedicate staff to
>>>>>>>>>>>>>>>>> improving Benchmark so that their products look better.
>>>>>>>>>>>>>>>>> Open source projects just cant do that, so we are at a
>>>>>>>>>>>>>>>>> distinct disadvantage.
>>>>>>>>>>>>>>>>> Should we allow a commercially driven OWASP project who's
>>>>>>>>>>>>>>>>> aim could be seen be to promote commercial software?
>>>>>>>>>>>>>>>>> If so, what sort of checks and balances does it need?
>>>>>>>>>>>>>>>>> Those are the sort of questions I'd like an independent
>>>>>>>>>>>>>>>>> review to look at.
>>>>>>>>>>>>>>>>> I do think there are some immediate steps that could be
>>>>>>>>>>>>>>>>> taken:
>>>>>>>>>>>>>>>>>    - I'd like to see the Benchmark project page clearly
>>>>>>>>>>>>>>>>>    state thats its at a very early stage and that the results are _not_ yet
>>>>>>>>>>>>>>>>>    suitable for use in commercial literature.
>>>>>>>>>>>>>>>>>    - I'd also like the main companies developing
>>>>>>>>>>>>>>>>>    Benchmark to be clearly stated on the main page. If and when other
>>>>>>>>>>>>>>>>>    companies get involved then this would actually help the project's claim of
>>>>>>>>>>>>>>>>>    vendor independence.
>>>>>>>>>>>>>>>>>    - And I'd love to see a respected co-leader added to
>>>>>>>>>>>>>>>>>    the project who is not associated with any commercial or open source
>>>>>>>>>>>>>>>>>    security tools:)
>>>>>>>>>>>>>>>>> And we should carry on discussing the project on this list
>>>>>>>>>>>>>>>>> - I think such discussions are very healthy, and I'd love to see this
>>>>>>>>>>>>>>>>> project mature to a state where it can be a trusted, independent and valued
>>>>>>>>>>>>>>>>> resource.
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias <
>>>>>>>>>>>>>>>>> tobias.gondrom at owasp.org> wrote:
>>>>>>>>>>>>>>>>>> @Simon:
>>>>>>>>>>>>>>>>>> yes, the leaders list is the place for your discussions
>>>>>>>>>>>>>>>>>> for project and chapter leaders
>>>>>>>>>>>>>>>>>> @Timo: I like your framing of "Don't ask what OWASP can
>>>>>>>>>>>>>>>>>> do for me, ask what I can do for OWASP."
>>>>>>>>>>>>>>>>>> That should and is indeed the spirit of OWASP:-)
>>>>>>>>>>>>>>>>>> Best regards, Tobias
>>>>>>>>>>>>>>>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>>>>>>>>>>>>>>>> I don't know enough about the matter to comment on this
>>>>>>>>>>>>>>>>>> case, but I feel that any situation where an OWASP project or any OWASP
>>>>>>>>>>>>>>>>>> initiative for that matter, is using OWASP to promote its own business
>>>>>>>>>>>>>>>>>> interests should be stopped.  We need to get rid of bad apples in
>>>>>>>>>>>>>>>>>> ...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20151127/29329223/attachment-0001.html>

More information about the Owasp-board mailing list