[Owasp-board] [Owasp-leaders] OWASP Benchmark project - potential conflict of interest

johanna curiel curiel johanna.curiel at owasp.org
Fri Nov 27 17:53:03 UTC 2015


*>>He has said that he is not responsible for Contrast's marketing team,
but that he would speak with the people who are.*

What about his own (Jeff) advertising on an article he wrote about this
promoting Contrast and Benchmark?

*>>In my experience, these kind of comments border on insults and only
cause folks to harden their opinions.*

*Agree.None of you have consider the fact that the project leader went and
pushed a project review so he could go and sell?Very unethical. So far this
is the first project I see doing something like this.*

This project should be set as incubator until it gains users. Just because
 is now polluted and contaminated with bad publicity , other vendors will
be more discouraged to support this project

*>>Once again I feel these gentlemen got away with a kind of brand abuse
that is very hurtful to the OWASP community but I am at a loss as to how
handle or prevent these kinds of mishaps - especially when board members
like yourself seem willing to - from what I see - brush it under the rug.*

ABSOLUTELY

*This is my official request to the Board: *
*Set the project back to incubator until it proves it has more people to
use it*

*Have you use it Josh? Honestly this project has ZERO automation and poor
error handling *
*Enough for a LAB, just roughly but no way has established itself and
gained users.*

*3 issues... 3*





On Fri, Nov 27, 2015 at 1:23 PM, Josh Sokol <josh.sokol at owasp.org> wrote:

> Admittedly, this was my gut reaction at first as well.  I began linking
> all of these companies, people, and projects together in my mind (there are
> some loose links there) and painted a big conspiracy picture similar to
> what Jim and Dinis have stated.  But, after speaking directly with Jeff,
> and hearing about the conversation that Dave and Matt had, I've changed my
> mind.
>
> I think it begins with the project itself.  If you aren't sold on the idea
> of the Benchmark, then you'll never be able to get to the same place.  My
> original line of thinking was that it was just a bar for vendors to compare
> their tools against eachother, but that's a bit myopic.  We are in an
> industry where things evolve very quickly.  As a customer of these tools, I
> know firsthand that something that a tool does today may not be the case a
> week from now.  Likewise, new features are being added daily and I need a
> point-in-time metric to be able to gauge continual effectiveness.  Cool,
> right?  But not a game changer.  The game changer part comes when you
> realize that by developing and evolving the tests that go into the
> Benchmark, we are moving the bar higher and higher.  We (OWASP) are
> effectively setting the standard by which these tools will be compared.  A
> tool that receives a lower score on the Benchmark today knows exactly what
> they need to work on in order to pass that test tomorrow and we already
> have examples of tools that have made improvements because of their
> Benchmark score (Ask Simon about ZAP's experience with the Benchmark).  I
> don't think that anyone can argue that the Benchmark project isn't being
> effective when OWASP's own tools are being driven forward as a result of
> using it.
>
> But, but, but, Dave and Jeff own Aspect and have stock in Contrast and
> Jeff is the Contrast CTO and Contrast got good scores so it's a conspiracy
> right?  Is there some code that allows Contrast to use the Benchmark?
> Absolutely.  Can you really blame Dave for starting his testing on the
> effectiveness of the Benchmark with a tool that he owned and is familiar
> with?  If I were going to start a similar project, there's no question in
> my mind that I would begin my testing with the tools that I have available
> to me.  That said, is there code that allows other tools to use the
> Benchmark?  Absolutely.
>
> Regarding "Dave has a history of breaching his duty to be vendor neutral",
> while I cannot comment on his past actions, I can judge what we've seen
> recently.  Matt saw a presentation from Dave on the Benchmark at a
> conference in Chicago.  He said that he felt that the message was
> appropriate and while IAST tools were mentioned as receiving higher scores,
> it wasn't a "Contrast is the best" type of message, more of a generality.
> I saw a very similar (if not the same) talk by Jeff at LASCON 2015 and the
> message was exactly the same.  I watched the talk expecting some sort of
> impropriety, but found none.  So, perhaps Dave has abused some privilege
> granted to him in the past, but what I've seen from him at this point, with
> respect to the Benchmark, has been appropriate.
>
> You have a very good point with respect to the Contrast marketing message
> around the Benchmark.  It's been completely absurd, over the top, and, in
> my personal opinion, intolerable.  In fact, I experienced the same thing
> that you talked about with them at LASCON 2015 where they stood in front of
> the door of the room Jeff was speaking in and scanned attendees as they
> went into the talk.  I agree that these types of aggressive marketing
> tactics cannot be tolerated at OWASP.  In addition, we have seen several
> marketing messages from them effectively implying that OWASP endorses
> Contrast.  Clearly this is not OK.  I've spoken with Jeff about it and we
> agreed that it is not in the Benchmark's best interest to have this
> aggressive Contrast marketing around it at such an early stage.  He has
> said that he is not responsible for Contrast's marketing team, but that he
> would speak with the people who are.  I haven't seen a single message from
> them since so I'm guessing that he's made good on this promise.  While
> that's an excellent start, OWASP's takeaway here should be that we need to
> do a better job with our brand usage guidelines both in terms of the
> wording and enforcement.  There are many other companies out there that use
> the OWASP brand and I think that we agree that selective enforcement
> against Contrast is not the right answer.  Paul and Noreen are actively
> working on this.  Either way, I think that implying that activities from a
> vendor's marketing department means that the project is not objective is
> not inappropriate.  If we feel that the project is not objective, then
> separate measures need to be taken to drive contribution diversity into
> it.  That I absolutely agree with and the message from Dave was that he
> would love to have more contributors to his project.  But, seeing as we
> cannot force people to work on it, this becomes a matter of "put up or shut
> up".  The same goes for the experts that you said reviewed the code.  If
> they feel that it is somehow skewed towards Contrast, they have the power
> to change that.  Now, if someone tries to participate and Dave tells them
> "No thanks", then I agree we have a problem, but I don't hear anyone
> inferring that happened.
>
> Please, let's drop the conspiracy theories and focus on the tangible
> things that we can do to help an OWASP project to be more successful.  Help
> find more participants to drive diversity, update our brand usage
> guidelines to prevent abuse, enforce them widely, etc.  Thank you.
>
> ~josh
>
> On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico <jim.manico at owasp.org> wrote:
>
>> Dinis,
>>
>> Like a rare celestial moment when all the planets plus Pluto are aligned,
>> I just read your email on the future of OWASP projects thinking, "Dinis is
>> spot on".
>>
>> Reflecting on projects I manage or work on...
>>
>> The Java Encoder and HTML Sanitizer are likely best moved to Apache now
>> that they have reached a measure of adoption and maturity. Apache would be
>> a much better long term custodian. Perhaps the same for AppSensor, but not
>> my project - just thinking out loud.
>>
>> Other similar defensive projects are still being noodled on, so OWASP is
>> a decent home for these research efforts.
>>
>> The whole tools category is also something to consider. Dependency Check
>> and of course ZAP are some of the best projects that OWASP offers, are they
>> best served where they are today? Both have rich communities of developers
>> but I don't see the foundation doing much to support these efforts.
>>
>> ASVS has the opportunity to effect massive change, I would to love to see
>> major investment and volunteer activity here. Pro tech writer, detailed
>> discourses on each individual requirement, etc. If I was king (and I am
>> not, at all) I would invest in ASVS on a 6 figure scale. (And who started
>> ASVS? Jeff, Dave and Boberski, hat tip to such a marvelous idea). Or maybe
>> moving ASVS to the W3C or IETF would help it grow?
>>
>> The Proactive Controls was a pet project but as we approach 2.0 we have
>> several active/awesome volunteers working on it. We will be making the doc
>> "world editable" to make contributions easy. OWASP seems like a good home
>> for such an awareness doc. Same with T10, especially if community edits are
>> welcome.
>>
>> Anyhow, I'm with you on this Dinis. Once a project starts to reach
>> production quality, spinning off the project as an external project or
>> moving it to a different foundation where managing production software or
>> formal standards is their thing seems realistic.
>>
>> I don't have all the answers here, but your email certainly resonated
>> with me.
>>
>> Aloha,
>> --
>> Jim Manico
>> Global Board Member
>> OWASP Foundation
>> https://www.owasp.org
>> Join me in Rome for AppSecEU 2016!
>>
>> On Nov 26, 2015, at 11:26 PM, Dinis Cruz <dinis.cruz at owasp.org> wrote:
>>
>> Jim's reading of this situation is exactly my view on the value of the
>> Contrast tool and how it has been 'pushing' the rules of engagement to an
>> very 'fuzzy' moral/ethical/commercial limit :)
>>
>> As per my last email, a key problem here is the 'perceived expectation'
>> of what is an OWASP project, and how it should be consumed.
>>
>> If you look at the OWASP benchmark as a research project, then the only
>> way it could be making the kind of claims it makes (and have credibility)
>> is if it had evolved from OWASP, with its own (diverse) community
>>
>> On 26 November 2015 at 21:01, Jim Manico <jim.manico at owasp.org> wrote:
>>
>>> I have a different take on this situation but my opinion is the
>>> "minority opinion". I will respect the rest of the boards take on this, but
>>> here is how I see it.
>>>
>>> First of all, Jeff has stated that he feels I am attacking him
>>> personally from a past personal grudge, and frankly I do not fault him for
>>> that perspective since we definitely have history with conflict. So it's
>>> fair to take my opinion on this with a grain of salt.
>>>
>>> I look at this situation from the perspective of a forensic investigator.
>>>
>>> 1) The Benchmark project had Contrast hooks and only Contrast hooks in
>>> it when I reviewed it so this leads me to believe that the project was
>>> clearly built with Contrast in mind from the ground up, at least in some
>>> way.
>>> 3) Dave has a history of breaching his duty to be vendor neutral. He was
>>> gifted with a keynote in South Korea a few years ago, and used that
>>> opportunity to discuss and pitch Contrast, on stage, during a keynote -
>>> with Contrast specific slides. This is just supporting evidence of his
>>> intention at OWASP to push Contrast in ways that I think are against the
>>> intentions and goals of our foundation.
>>> 3) Other experts have reviewed the project and felt that many of the
>>> tests were very slanted and almost contrived to support Contrast. I can
>>> drag those folks into this conversation, but I do not think that would help
>>> in any way. So it's fair to call this point heresy.
>>> 4) I do not see this project as revolutionary, at all. Every vendor has
>>> their own test suite tuned for their tool. As the benchmark stands today, I
>>> see it as just another vendors product-specific benchmark. Mass
>>> collaboration from many vendors is not just a "nice to have" but a base
>>> requirement to get even close to useful for objective tool measurement.
>>> 5) Jeff stating that his Marketing people went over the line is also an
>>> admission that - well, they went over the line. By the same token Jeff was
>>> in his booth at AppSec USA surrounded by benchmark marketing material,
>>> discussing this to prospects and he even asked me and Mr Coates to wade
>>> into this debate and support Dave. So to say he was not involved and it was
>>> only his marketing people seems a stretch at best.
>>> 6) The Contrast marketing team was wandering around the conference
>>> zapping folks to get leads, and I asked them to stay in their booth, which
>>> is standard conference policy. These folks know better but are again going
>>> over the line to sell product at OWASP. There is a better way (like
>>> focusing on product capability and language support, have consistent +
>>> stellar customer service, have a humble and gracious attitude to all
>>> prospects and customers, actively participate in OWASP in a vendor neutral
>>> and community supportive way, etc).
>>>
>>> Please note, I think Contrast is a decent tool, I've offered to resell
>>> in the past, and I have recommended it in certain situations - even after
>>> this situation arose. I'm stating this out of honestly and desire to put my
>>> cards on the table. I truly want Jeff and Dave to be successful. They have
>>> dedicated their lives to AppSec and if anyone should win big-time, I hope
>>> it's them. I even told Jeff I hope he hits the mother load and donates a
>>> little back to OWASP.
>>>
>>> However, my instinct and evidence tell me that they both went over the
>>> line in the use of the OWASP brand to sell product.
>>>
>>> Now, Jeff makes a good point. We as a board and staff are very poor at
>>> enforcing brand management policy and it's not fair to single out Contrast,
>>> when many other vendors violate the brand, IMO. Just google OWASP and watch
>>> the ads fly that use the OWASP name to sell product.
>>>
>>> Also, any and every request that was made of Dave to adjust the project
>>> for the sake of vendor neutrality was taken very seriously. Regardless of
>>> Daves past intentions, he is clearly trying to do the right thing moving
>>> forward.
>>>
>>> I look to "postels principle" in this situation (this is otherwise known
>>> as the "robustness principle" and dates back to the creation of TCP) . This
>>> is paraphrased as, "Be liberal in what you take from others but be
>>> conservative in what you dish out". So I think it's critical that OWASP and
>>> any OWASP resource present itself in a strict vendor neutral way. But
>>> unless OWASP wants to be much more "even" in the enforcement of brand
>>> policy across the board to all violators, we should be fairly lax in the
>>> enforcement of these issues from the outside world.
>>>
>>> I am trying to be objective here. My trigonometry teacher once told me
>>> "I'd fail my mother" when I asked him if he would ever fail me (I was an A
>>> student). If my mother owned a security company and tried the same stunt,
>>> I'd have the same opinions about her actions as well.
>>>
>>> So what next? Well hello from the other side. I'm going back to
>>> listening to Adele's new album where I can sit in my deep feelings and
>>> reflect upon what the OWASP foundation has done to enrich my life. I would
>>> much rather keep out of this (and any other conflict laden situation at
>>> OWASP), but I feel it's my responsibility to speak up.
>>>
>>> Aloha,
>>> --
>>> Jim Manico
>>> Global Board Member
>>> OWASP Foundation
>>> https://www.owasp.org
>>> Join me in Rome for AppSecEU 2016!
>>>
>>> On Nov 26, 2015, at 9:09 PM, Josh Sokol <josh.sokol at owasp.org> wrote:
>>>
>>> I would be happy to provide an update.
>>>
>>>    - Matt Konda and Dave Wichers, the Benchmark Project Leader, had a
>>>    conversation a few weeks back.  To summarize their conversation, Dave
>>>    acknowledges the currently lack of diversity in his project and it is his
>>>    sincere desire to drive more people to it to help.  He also acknowledges
>>>    the issues with Contrast's extreme marketing around the project and feels
>>>    that it is in everyone's best interests for them to curb it back.  While he
>>>    does have an ownership stake in Contrast, he works at Aspect and has no
>>>    control over the marketing messages that they are putting out there.  From
>>>    the Board perspective, there has been no evidence of any impropriety on
>>>    Dave's part and it should be our goal to drive more diversity into the
>>>    project to support Dave.  Dave appears to be sincere in his desires to
>>>    create a tool where OWASP can tell vendors what we expect from their
>>>    tools.  If the main issue is that only members of Aspect are working on it,
>>>    then the best thing that we can do is try to get him some outside
>>>    assistance.  We are also asking that the project be opened up to commits
>>>    via Git so that outsiders can push commits to it.
>>>    - Josh Sokol and Jeff Williams, the CTO of Contrast, had a
>>>    conversation a few weeks back.  To summarize their conversation, Jeff
>>>    believes that the work that Dave is doing on the Benchmark is a game
>>>    changer in that it gives OWASP the power in dictating what these tools need
>>>    to be finding.  He wants the Benchmark to be successful and understands
>>>    that it needs to be diverse in order to be trusted.  He recognizes that
>>>    Dave is trying to do that and does not want the marketing message from
>>>    Contrast to interfere with his efforts.  Jeff felt that the "Lab" status
>>>    granted to Benchmark meant that it was ready for mainstream adoption, that
>>>    it had 21k tests, and was almost a year old, and didn't see anything wrong
>>>    with marketing their results, but has agreed to talk to their marketing
>>>    team to get them to lay off that message for now.  From the Board
>>>    perspective, we have come to the realization that our brand usage
>>>    guidelines need an overhaul to clarify what is and is not allowed.  We have
>>>    made a few proposals and have reached out to Mozilla to gain more insight
>>>    on their guidelines and even ask for assistance.  Noreen and Paul are
>>>    taking lead on these efforts.
>>>    - There is a note in the notes that the Board was supposed to follow
>>>    up with an open letter to the community and companies involved describing
>>>    our review and actions.  I don't think that has happened so I will remind
>>>    the person who took on that action item.
>>>
>>> I'm happy to answer any questions that you may have.
>>>
>>> ~josh
>>>
>>> On Thu, Nov 26, 2015 at 11:55 AM, Tobias <tobias.gondrom at owasp.org>
>>> wrote:
>>>
>>>> There have been several conversations on that matter and a dedicated
>>>> call. Unfortunately for personal reasons I could not attend the last call
>>>> as it was at 04:00am my local time, but all other board members did
>>>> participate.
>>>>
>>>> Could please one of my fellow board members give an update.
>>>>
>>>> Best, Tobias
>>>>
>>>>
>>>>
>>>> On 26/11/15 18:04, Timo Goosen wrote:
>>>>
>>>> I would also like to know the answer to Simon's question. We need to
>>>> get rid of bad apples in OWASP in my opinion, there are too many people
>>>> just using the OWASP "name" or "brand" to improve their own financial
>>>> situation or career.
>>>>
>>>> Regards.
>>>> Timo
>>>>
>>>> On Thu, Nov 26, 2015 at 1:13 PM, psiinon <psiinon at gmail.com> wrote:
>>>>
>>>>> Paul, and the rest of the board,
>>>>>
>>>>> Its been over 2 months since I raised this issue.
>>>>> Whats happening?
>>>>> Has the board even discussed it?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>> On Tue, Oct 20, 2015 at 10:00 PM, Paul Ritchie <paul.ritchie at owasp.org
>>>>> > wrote:
>>>>>
>>>>>> Eoin, Johanna, All:
>>>>>>
>>>>>> In an earlier email, Josh Sokol mentioned that he will be speaking in
>>>>>> the next day or 2 to their CTO, while at LASCON, as a representative of the
>>>>>> OWASP Board.  Following that feedback, the Board has action to take the
>>>>>> next steps.
>>>>>>
>>>>>> Just an FYI that all comments are recognized and action is being
>>>>>> taken.
>>>>>>
>>>>>> Paul
>>>>>>
>>>>>>
>>>>>>
>>>>>> Best Regards, Paul Ritchie
>>>>>> OWASP Executive Director
>>>>>> paul.ritchie at owasp.org
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 20, 2015 at 1:54 PM, johanna curiel curiel <
>>>>>> johanna.curiel at owasp.org> wrote:
>>>>>>
>>>>>>> Time for owasp to do a public statement and put a clear story
>>>>>>> regarding this abusive behavior of Owasp brand
>>>>>>>
>>>>>>>
>>>>>>> On Tuesday, October 20, 2015, Eoin Keary <eoin.keary at owasp.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Folks,
>>>>>>>>
>>>>>>>> The project should be immediately shelved it's simply bad form.
>>>>>>>>
>>>>>>>> This is damaging to OWASP, the industry and exactly what OWASP is
>>>>>>>> not about.
>>>>>>>>
>>>>>>>> There is a clear conflict of interest and distinct lack of science
>>>>>>>> behind the claims made by Contrast.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Eoin Keary
>>>>>>>> OWASP Volunteer
>>>>>>>> @eoinkeary
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel <
>>>>>>>> johanna.curiel at owasp.org> wrote:
>>>>>>>>
>>>>>>>> At the moment we did the project review, we observed that the
>>>>>>>> project did not have enough testing to be considered in any form as 'ready'
>>>>>>>>  for benchmarking, neither that it had yet the community adoption, however
>>>>>>>> technically speaking as it has been classified by the leaders, the project
>>>>>>>> is at the beta stage.
>>>>>>>>
>>>>>>>> Indeed , Dave had the push to have the project reviewed but it was
>>>>>>>> never clear that later on the project was going to be advertisied this way.
>>>>>>>> That all happend after the presentation at Appsec.
>>>>>>>>
>>>>>>>> I had my concerns regarding how sensitive is the subject of the
>>>>>>>> project ,but I think we should allow project leaders to develop their
>>>>>>>> communication strategy even if this has conflict of interest. It all
>>>>>>>> depends how they behave and how they manage this.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tuesday, October 6, 2015, Michael Coates <
>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>
>>>>>>>>> It's not really that formal to add to the agenda, just a wiki that
>>>>>>>>> we add in the text.
>>>>>>>>>
>>>>>>>>> I think you can safely assume it will get the appropriate
>>>>>>>>> discussion.
>>>>>>>>>
>>>>>>>>> On Oct 6, 2015, at 7:16 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Really?? Its not on the agenda yet for the next meeting??
>>>>>>>>> How does it get added to the agenda?
>>>>>>>>> And that was a formal request if that makes any difference :)
>>>>>>>>> I'm all in favour of getting the facts straight before any actions
>>>>>>>>> are taken, hence my request for an 'ethical review' or whatever it should
>>>>>>>>> be called.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Simon
>>>>>>>>>
>>>>>>>>> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates <
>>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>>
>>>>>>>>>> First step is to get all of our information straight so we're
>>>>>>>>>> clear on where things are at.
>>>>>>>>>>
>>>>>>>>>> This was not on the board agenda last meeting and is also not on
>>>>>>>>>> the next agenda as of yet (of course it could always be added if needed).
>>>>>>>>>>
>>>>>>>>>> We are aware that people have raised questions though.   I'm
>>>>>>>>>> hoping we can get a clear understanding of all the facts and then discuss
>>>>>>>>>> if changes are needed.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hey Michael,
>>>>>>>>>>
>>>>>>>>>> Is the board going to take any action?
>>>>>>>>>> Were there any discussions about this controversy in the board
>>>>>>>>>> meeting at AppSec USA?
>>>>>>>>>> If not will it be on the agenda for the meeting on October 14th?
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Simon
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates <
>>>>>>>>>> michael.coates at owasp.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Simon
>>>>>>>>>>>
>>>>>>>>>>> I posted the below message earlier today. At this point my goal
>>>>>>>>>>> is to just gain clarity over the current reality and ideally drive to a
>>>>>>>>>>> shared state of success. This message doesn't seem to be reflected in the
>>>>>>>>>>> list yet. It could be because my membership hasn't been approved or because
>>>>>>>>>>> of mail list delays (I miss Google groups). But I think these questions
>>>>>>>>>>> will start the conversation.
>>>>>>>>>>>
>>>>>>>>>>> (This was just me asking questions as a curious Owasp member,
>>>>>>>>>>> not any action on behalf of the board)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>
>>>>>>>>>>> *From:* Michael Coates <michael.coates at owasp.org>
>>>>>>>>>>> *Date:* October 5, 2015 at 6:20:23 PM PDT
>>>>>>>>>>> *To:* owasp-benchmark-project at lists.owasp.org
>>>>>>>>>>> *Subject:* *Project Questions*
>>>>>>>>>>>
>>>>>>>>>>> OWASP Benchmark List,
>>>>>>>>>>>
>>>>>>>>>>> I've heard more about this project and am excited about the idea
>>>>>>>>>>> of an independent perspective of tool performance. I'm trying to understand
>>>>>>>>>>> a few things to better respond to questions from those in the security &
>>>>>>>>>>> OWASP community.
>>>>>>>>>>>
>>>>>>>>>>> In my mind there are two big areas for consideration in a
>>>>>>>>>>> benchmark process.
>>>>>>>>>>> 1. Are the benchmarks testing the right areas?
>>>>>>>>>>> 2. Is the process for creating the benchmark objective & free
>>>>>>>>>>> from conflicts of interest.
>>>>>>>>>>>
>>>>>>>>>>> I think as a group OWASP is the right body to align on #1.
>>>>>>>>>>>
>>>>>>>>>>> I'd like to ask for some clarifications on item #2. I think it's
>>>>>>>>>>> important to avoid actual conflict of interest and also the appearance of
>>>>>>>>>>> conflict of interest. The former is obvious why we mustn't have that, the
>>>>>>>>>>> latter is critical so others have faith in the tool, process and outputs of
>>>>>>>>>>> the process when viewing or hearing about the project.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 1) Can we clarify whether other individuals have submitted
>>>>>>>>>>> meaningful code to the project?
>>>>>>>>>>> Observation:
>>>>>>>>>>> Nearly all the code commits have come from 1 person (project
>>>>>>>>>>> lead).
>>>>>>>>>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>>>>>>>>>>
>>>>>>>>>>> 2) Can we clarify the contributions of others and their
>>>>>>>>>>> represented organizations?
>>>>>>>>>>> Observation:
>>>>>>>>>>> The acknowledgements tab listed two developers (Juan Gama & Nick
>>>>>>>>>>> Sanidas) both who work at the same company as the project lead. It seems
>>>>>>>>>>> other people have submitted some small amounts of material, but overall it
>>>>>>>>>>> seems all development has come from the same company.
>>>>>>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>>>>>>>>
>>>>>>>>>>> 3) Can we clarify in what ways we've mitigated the potential
>>>>>>>>>>> conflict of interest and also the appearance of a conflict of interest?
>>>>>>>>>>> This seems like the largest blocker for wide spread acceptance of this
>>>>>>>>>>> project and the biggest risk.
>>>>>>>>>>> Observation:
>>>>>>>>>>> The project lead and both of the project developers works for a
>>>>>>>>>>> company with very close ties to one of the companies that is evaluated by
>>>>>>>>>>> this project. Further, it appears the company is performing very well on
>>>>>>>>>>> the project tests.
>>>>>>>>>>>
>>>>>>>>>>> 4) If we are going to list tool vendors then I'd recommend
>>>>>>>>>>> listing multiple vendors for each category.
>>>>>>>>>>> Observation:
>>>>>>>>>>> The tools page only lists 1 IAST tool. Since this is the point
>>>>>>>>>>> of the potential conflict of interest it is important to list numerous IAST
>>>>>>>>>>> tools.
>>>>>>>>>>>
>>>>>>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Tool_Support_2FResults
>>>>>>>>>>>
>>>>>>>>>>> 5) Diverse body with multiple points of view
>>>>>>>>>>> Observation:
>>>>>>>>>>> There is no indication that multiple stakeholders are present to
>>>>>>>>>>> review and decide on the future of this project. If they exist, a new
>>>>>>>>>>> section should be added to the project page to raise awareness. If they
>>>>>>>>>>> don't exist, we should reevaluate how we are obtaining an independent view
>>>>>>>>>>> of the testing process.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Again, I think the idea of the project is great. From my
>>>>>>>>>>> perspective clarifying these questions will help ensure the project is not
>>>>>>>>>>> only objective, but also perceived as objective from someone reviewing the
>>>>>>>>>>> material. Ultimately this will contribute to the success and growth of the
>>>>>>>>>>> project.
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Michael Coates
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> OK, based on the concerns raised so far I think the board should
>>>>>>>>>>> initiate a review of the OWASP Benchmark project.
>>>>>>>>>>> I'm not raising a formal complaint against it, I'm just
>>>>>>>>>>> requesting a review.
>>>>>>>>>>> And I dont think it needs a 'standard' project review - Johanna
>>>>>>>>>>> has already done a very good job of this.
>>>>>>>>>>> Not sure what sort of review you'd call it, I'll leave the
>>>>>>>>>>> naming to others :)
>>>>>>>>>>>
>>>>>>>>>>> I'm concerned that we have an OWASP project lead by a company
>>>>>>>>>>> who has a clear commercial stake in the results.
>>>>>>>>>>> Bringing more companies on board will help, but I'm still not
>>>>>>>>>>> sure that alone will make it independent enough.
>>>>>>>>>>> Commercial companies can afford to dedicate staff to improving
>>>>>>>>>>> Benchmark so that their products look better.
>>>>>>>>>>> Open source projects just cant do that, so we are at a distinct
>>>>>>>>>>> disadvantage.
>>>>>>>>>>> Should we allow a commercially driven OWASP project who's aim
>>>>>>>>>>> could be seen be to promote commercial software?
>>>>>>>>>>> If so, what sort of checks and balances does it need?
>>>>>>>>>>> Those are the sort of questions I'd like an independent review
>>>>>>>>>>> to look at.
>>>>>>>>>>>
>>>>>>>>>>> I do think there are some immediate steps that could be taken:
>>>>>>>>>>>
>>>>>>>>>>>    - I'd like to see the Benchmark project page clearly state
>>>>>>>>>>>    thats its at a very early stage and that the results are _not_ yet suitable
>>>>>>>>>>>    for use in commercial literature.
>>>>>>>>>>>    - I'd also like the main companies developing Benchmark to
>>>>>>>>>>>    be clearly stated on the main page. If and when other companies get
>>>>>>>>>>>    involved then this would actually help the project's claim of vendor
>>>>>>>>>>>    independence.
>>>>>>>>>>>    - And I'd love to see a respected co-leader added to the
>>>>>>>>>>>    project who is not associated with any commercial or open source security
>>>>>>>>>>>    tools:)
>>>>>>>>>>>
>>>>>>>>>>> And we should carry on discussing the project on this list - I
>>>>>>>>>>> think such discussions are very healthy, and I'd love to see this project
>>>>>>>>>>> mature to a state where it can be a trusted, independent and valued
>>>>>>>>>>> resource.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Simon
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias <tobias.gondrom at owasp.org
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> @Simon:
>>>>>>>>>>>> yes, the leaders list is the place for your discussions for
>>>>>>>>>>>> project and chapter leaders
>>>>>>>>>>>> @Timo: I like your framing of "Don't ask what OWASP can do for
>>>>>>>>>>>> me, ask what I can do for OWASP."
>>>>>>>>>>>> That should and is indeed the spirit of OWASP:-)
>>>>>>>>>>>> Best regards, Tobias
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I don't know enough about the matter to comment on this case,
>>>>>>>>>>>> but I feel that any situation where an OWASP project or any OWASP
>>>>>>>>>>>> initiative for that matter, is using OWASP to promote its own business
>>>>>>>>>>>> interests should be stopped.  We need to get rid of bad apples in OWASP.
>>>>>>>>>>>>
>>>>>>>>>>>> OWASP is becoming a brand if you would like to think of it that
>>>>>>>>>>>> way and we are going to see many more cases of people trying to use OWASP
>>>>>>>>>>>> to spread their business interests. At the end of the day everyone should
>>>>>>>>>>>> be acting with an attitude of:"Don't ask what OWASP can do for me, ask what
>>>>>>>>>>>> I can do for OWASP?"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Regards.
>>>>>>>>>>>> Timo
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 30, 2015 at 11:48 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> So, a load of controversy about OWASP Benchmark on twitter,
>>>>>>>>>>>>> but no discussion on the leaders list :(
>>>>>>>>>>>>> Is this now the wrong place to discuss OWASP projects??
>>>>>>>>>>>>>
>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Sep 24, 2015 at 10:36 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi folks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've got some concerns about the OWASP Benchmark project.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I _like_ benchmarks, and I'm very pleased to see an active
>>>>>>>>>>>>>> OWASP project focused on delivering one.
>>>>>>>>>>>>>> I think the project has some technical limitations, but thats
>>>>>>>>>>>>>> fine given the stage the project is at, ie _very_ early.
>>>>>>>>>>>>>> I dont think that any firm conclusions should be drawn from
>>>>>>>>>>>>>> it until its been significantly enhanced.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My concerns are around the marketing that one of the
>>>>>>>>>>>>>> companies sponsoring the Benchmark project has started using.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here we have a company that leads an OWASP project that just
>>>>>>>>>>>>>> happens to show that their offering in this area appears to be
>>>>>>>>>>>>>> _significantly_ better than any of the competition.
>>>>>>>>>>>>>> Their recent press release stresses that its an OWASP
>>>>>>>>>>>>>> project, make the most of the fact that
>>>>>>>>>>>>>>
>>>>>>>>>>>>> ...
>
> [Message clipped]
> _______________________________________________
> Owasp-board mailing list
> Owasp-board at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-board
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20151127/ad9a7bf1/attachment-0001.html>


More information about the Owasp-board mailing list