[Owasp-leaders] [Owasp-board] OWASP Benchmark project - potential conflict of interest

Dave Wichers dave.wichers at owasp.org
Tue Dec 1 03:37:12 UTC 2015


Thanks so much for your post. For clarification, Justin refers to Coverity
numerous times throughout this post, but really means Contrast.


On 11/30/15, 4:55 PM, "Justin Searle" <justin at meeas.com> wrote:

>Psiinon, out of curiosity, from a purely project/code perspective,
>what do you feel would make the Benchmark tool more "independent"?
>From digging around in the source code, it seems they already have a
>fair number of report parsers for several OSS and commercial tools in
>their tree.  To keep this thread a bit more clean, perhaps post your
>response in the Benchmark project mailling list or in a GitHub issue
>then reply to this thread with link.  That will allow the Benchmark
>team to directly receive that feedback and allow those of use
>interested to follow up there to have the needed discussions to
>improve the tool.
>Eoin, Dinis, and Jim, it has been a while.  Sorry I've been so removed
>from this community the last couple of years.  From reading through
>the thread and articles, I don't think it is fair to say the board has
>been sitting on the issue.  From Josh's comments, it seems like the
>board did take action on this, however I understand if you disagreed
>with their decision.  You have always been some of the most vocal
>leaders here, and I applaud that.  Your contributions have been great
>to OWASP over the years, so please keep it up!
>Most of the concerns I've seen stated are around use of the OWASP
>brand in marketing.  I personally do not see merit in many of the
>concerns, however my personal greatest concerns are around statements
>like this:
>    "OWASP reports that the best static analysis tools score in the
>low 30¹s (out of 100) against this benchmark."
>That was in the Dark Reading article, and other statements were made
>like that in the Twitter video posted above.  That I feel is the most
>egregious of the brand misuse as it implies that OWASP as an
>organization has formally made statement.  Keep the full project name
>in tact such as "The OWASP Benchmark tool reports..." would be a much
>more accurate and less brand abusive way to make that statement, and
>based on current OWASP policies, much more inline with what is
>permitted.  While OWASP members feel even that is going too far,
>please remember that OWASP does not own the vast majority of the OWASP
>projects.  Each OWASP project is usually owned by the author and in
>many cases, any contributor to that project.  OWASP as an legal entity
>owns a very small percentage (none that I can even name off the top of
>my head).
>A few years ago when we were initially working on the new project
>leaders handbook and project roadmap (before OWASP disbanded all the
>global committees in 2013, including the Global Projects Committee),
>we discussed chaining the official verbiage to and OWASP "sponsored"
>project.  I think it is unfortunate we didn't codify that terminology,
>but regardless, I think that is the always a good way to think of
>OWASP projects: projects owned and run by individuals of the
>community.  However, not everyone in the OWASP community understand
>this.  This is easy to see in such statements as "Allowing this
>project to exist without ..." and forcing a "project be opened up to
>commits via Git so that outsiders can push commits to it" and OWASP
>should "decide on the future of this project".  Personally, I think
>the most of the drama around OWASP projects comes from this
>misunderstanding and OWASP community members trying to manage an OWASP
>project that OWASP doesn't own.
>However one of the most difficult issues that perpetuates this problem
>and in many cases conflicts concerning brand abuse is project naming.
>Since OWASP currently allows projects to use the OWASP name in their
>project name (which I think is a mistake), it is hard to refer to a
>project without in some way evoking the OWASP brand.  There is very
>little legal recourse in most countries to state a fact that "tool X
>generated score Y for product Z" in their marketing literature.  That
>does not imply that tool X promotes product Z.  And if tool X happens
>to be named OWASP Benchmark, then that is not brand infringement in
>most countries.  If this is a concern to the OWASP community, then the
>better recourse would be to reconsider OWASP's permission to allow
>projects to use OWASP in their project name.
>Preventing people from making such statements is usually handled in a
>EULA saying how you can and can't use the tool and the tool output,
>which in most cases including this would be contrary from the official
>OSS definition and most OSS licenses.  So the best distance I think we
>could hope to obtain is to disallow the use of OWASP in any project
>And by the way, why would we ever want to stop ANY company out there
>from using OWASP tools and documentation?  Why would we ever want ANY
>company NOT to advertise that they use OWASP tools and documentation?
>Why would we NOT want a company to state they they use OWASP tools in
>their marketing literature?  As long as it is clear that OWASP does
>not endorse that company, we should encourage the spread and use of
>OWASP tools.  Does anyone have a problem with saying website Z has
>been tested for all OWASP Top 10 risks in their marketing literature?
>What about saying that all vulnerabilities identified by OWASP Zed
>Attack Proxy Project have been remediated in website Z literature?
>What about all the current DAST/SAST tools that have an "OWASP Top 10"
>testing mode?  I don't think any of these imply that a project is
>endorsed by OWASP, but if this is a concern for people, would ..."all
>vulnerabilities identified by Zed Attack Proxy Project have been
>remediated" be better?
>As for a Jeff (or his company) using the benchmark scores from his own
>OWASP sponsored project in marketing literature to help customers
>understand their commercial offering, I have no qualms with that.  I
>don't find that a breach of trust or brand abuse.  I only see brand
>abuse in statements mentioned above that stated "Owasp found..." and
>such where the tool name was not used, which is explicitly stating a
>false OWASP perspective.  Jeff and Coverity in benchmarking their tool
>against their own opensource project simply ties the two together in
>such a way that can be tested.  Based on statements made by Psiinon
>and others, including Coverity's competition, the tool works and does
>not seem to be skewed towards Coverity's tool, even though they score
>the highest.  The tool is opensource.  If anyone believes the tool
>unfairly scores Coverity's tool, or doesn't not provide benefit to
>other assessment tools who want to improve their scanning engines,
>please dig through the code and identify how it does that.  All I've
>seen so far is people disagreeing with how the metric is generated and
>the number of tests involved, which in itself doesn't seem to portray
>bias for one tool over another.  If the tool is found to favor
>Coverity's scanning tool, then that will be shown by someone with time
>and interest, and if that is the case, the brand loss will by
>Coverity's, not OWASP's.
>As for actions, I agree with the actions the board seems to have made
>so far.  I do not think any penalties be levied against the Benchmark
>project.  I do not think that they should be downgrade back to
>incubator, which seems a petty and meaningless action to me.  The
>maturity seems to say Lab quality more than many other existing Lab
>projects.  I do not think OWASP has any right or reason to force the
>Benchmark project to allow commits from additional persons.  Having a
>single person do actual commits to main trunks while other offer pull
>requests is common and very standard in OSS, and in now way portrays
>how "open" the project community is.  And banning companies from any
>mention OWASP projects in marketing efforts, wether project leaders
>are associate with said companies or not, would be foolish in our
>efforts to growing OWASP brand, as long as such marketing efforts do
>not implicitly or explicitly imply OWASP endorsement of a company, its
>tools, or its services.
>As for my suggestions to the OWASP board, I'd recommend the following:
> - An official statement on the Benchmark project page at owasp.org
>stating as Johanna suggested, that OWASP does not endorse any company,
>commercial tool, or commercial service
> - A request to Coverty to make a similar statement on their website
>and future marketing efforts just to clarify this misunderstanding
> - A formal cease and desist letter to Coverty to stop making explicit
>claims in OWASP's behalf such as "OWASP found ..." and to restrict all
>use of the term "OWASP" as part of the "OWASP Benchmark" project's
>formal name.
>As for the Benchmark project, I'd recommend the following:
> - If the tool doesn't already do so, I'd recommend a simply statement
>in the Benchmark reports saying that scores from the tool does not
>imply any endorsement for or against any tool tested.
> - Also, digging through your sourcecode tree, I noticed there doesn't
>seem to be any copyright notices.  I'd recommend you adding those
>copyright notices to whoever owns the code, otherwise it is hard to
>enforce any copyright license restrictions.
>And finally, as for the OWASP community, I'd encourage you to decide
>if it makes sense to remove the ability of projects to formally use
>OWASP as part of their project name.  If we don't do this then
>individual project brand and OWASP brand becomes commingled, and
>ownership becomes less clear.  Project leaders, for your current or
>future projects, I'd personally recommend you don't use OWASP in the
>title so you can build project brand recognition independent of OWASP,
>and instead do something like "Project X, an OWASP project" in your
>project marketing.
>Justin Searle
>Managing Partner - UtiliSec
>+1 801-784-2052
>justin at utilisec.com
>justin at meeas.com
>On Mon, Nov 30, 2015 at 10:40 AM, psiinon <psiinon at gmail.com> wrote:
>>> In the short term, It is better to remove it from OWASP, leaving the
>>> open for its return (in a future when some of the independence and
>>> issues have been solved)
>> I would be delighted to welcome an independent, high quality Benchmark
>> project back into OWASP :)
>>> Specially when recently we made David Rook remove this much more benign
>>> 'commercial content' from OWASP
>>> Dinis
>>> On 30 November 2015 at 17:17, psiinon <psiinon at gmail.com> wrote:
>>>> I'd like to start by saying that I actually _like_ the Benchmark
>>>> Myself and other ZAP developers have made some contributions to it,
>>>> we have used (and will continue to use) it to make ZAP better.
>>>> I think these sort of testing applications are very valuable to all
>>>> security tools, and I'd like to thank Dave and his team for the
>>>> amount of effort involved in developing and open sourcing it.
>>>> But I dont think it should be an OWASP project.
>>>> I do not think that a vendor led project can ever objectively evaluate
>>>> competing commercial and open source projects.
>>>> I do not think that just saying 'pull requests welcomed' makes a
>>>> vendor neutral.
>>>> I do not think that a project as mired in controversy as the Benchmark
>>>> project can ever recover to become truly independent.
>>>> I am very disappointed in the Boards handling of this affair.
>>>> Ideally I'd like Dave to understand how much damage this project has
>>>> and to withdraw it as an OWASP project, while still maintaining it as
>>>>a very
>>>> valuable vendor led open source resource.
>>>> Failing that I really hope that the Board comes to its senses and
>>>> the Benchmark project before even more damage is done.
>>>> At the _very_ least it should flag the project as being 'in dispute'
>>>> Kevin suggested) while a more detailed evaluation is performed.
>>>> However I'm rapidly loosing loosing faith that the Board will do the
>>>> right thing and protect OWASP's image in the way that they should have
>>>> already done.
>>>> Members - please make your voices heard before more people and
>>>> leave OWASP.
>>>> Simon
>>>> On Sat, Nov 28, 2015 at 5:14 AM, Jim Manico <jim.manico at owasp.org>
>>>>> WAFEC does not "do vendor assessment"; they define a comprehensive
>>>>> standard built by many vendors and let the community use that
>>>>>standard to
>>>>> measure tools on their own. Just a FYI, I was involved in the early
>>>>> of this project. (Things may have changed since my involvement, I'm
>>>>> Tony has more details here)
>>>>> Johanna's comments on this issue lead me to believe that the damage
>>>>> to both OWASP and DHS is even more destructive that I thought. It
>>>>>saddens me
>>>>> to see this level of abuse just to sell product.
>>>>> --
>>>>> Jim Manico
>>>>> Global Board Member
>>>>> OWASP Foundation
>>>>> https://www.owasp.org
>>>>> Join me in Rome for AppSecEU 2016!
>>>>> On Nov 28, 2015, at 2:40 AM, Josh Sokol <josh.sokol at owasp.org> wrote:
>>>>> One of the ideas that Andrew proposed was actually approaching WAFEC
>>>>> learn more about how they do vendor assessment in a neutral way.
>>>>>It's great
>>>>> to hear that we have a resource here already that we can leverage.
>>>>>I wasn't
>>>>> aware of your affiliation.
>>>>> ~josh
>>>>> On Nov 27, 2015 2:47 PM, "Tony Turner" <tony.turner at owasp.org> wrote:
>>>>>> I sincerely hope so. That's not the impression I got from others
>>>>>> comments. Personally I haven't used the tool at all, but as I'm the
>>>>>> lead for another product evaluation project (WAFEC) I'm very
>>>>>>sensitive to
>>>>>> the need of collaboration with many different vendors. There really
>>>>>>has to
>>>>>> be a very high level (almost paranoid level) transparency with how
>>>>>> are approached, worked with, how requirements for evaluation are
>>>>>> and how metrics are derived.
>>>>>> It appears the project team is attempting to address these last 2
>>>>>> somewhat but I'd like to see more specifics, and the lack of
>>>>>>information on
>>>>>> how they are addressing vendor communication, participation and
>>>>>> seems a bit concerning. Lastly, it is my opinion that project
>>>>>> should not belong to anyone working for or with a
>>>>>> stake for any vendor being evaluated. I think this is a flawed
>>>>>>model and
>>>>>> should transition to a vendor neutral party.
>>>>>> On Nov 27, 2015 3:16 PM, "Josh Sokol" <josh.sokol at owasp.org> wrote:
>>>>>>> I don't know what qualifies as "significant" in your mind, but my
>>>>>>> understanding is that there have been contributions from other
>>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>>>> Still, Dave would like more, but he can't force them to help.
>>>>>>> ~josh
>>>>>>> On Fri, Nov 27, 2015 at 1:45 PM, Tony Turner
>>>>>>><tony.turner at owasp.org>
>>>>>>> wrote:
>>>>>>>> While I can appreciate that they started with Contrast, if there
>>>>>>>> hasn't been significant effort to include other vendors it's a
>>>>>>>> benchmark. It's easy to state you haven't gotten support from
>>>>>>>>other vendors
>>>>>>>> and that's fine, but until you do there's really nothing to
>>>>>>>>release. Why was
>>>>>>>> it ever upgraded? Talking about the results without an accurate
>>>>>>>> analysis is akin to snake oil.
>>>>>>>> On Nov 27, 2015 1:49 PM, "Josh Sokol" <josh.sokol at owasp.org>
>>>>>>>>> Thank you for the links to those articles.  The first one
>>>>>>>>> the strengths and weaknesses of the different methods of
>>>>>>>>>evaluating for
>>>>>>>>> application vulnerabilities.  The section on the Benchmark seems
>>>>>>>>> appropriate to me.  That seems like an excellent description of
>>>>>>>>>what the
>>>>>>>>> project is designed to do.  I see some metrics in there about
>>>>>>>>>which tools
>>>>>>>>> are more effective on which types of vulnerabilities, but I
>>>>>>>>>don't see him
>>>>>>>>> straight up saying "The OWASP Benchmark proves that Contrast is
>>>>>>>>> This seems like statements made based on some level of testing
>>>>>>>>>and research.
>>>>>>>>> Honestly, I don't see any OWASP brand abuse in that article.
>>>>>>>>>Whether it's
>>>>>>>>> in good taste or not at this stage in the project is certainly
>>>>>>>>> but if you look at the brand usage guidelines
>>>>>>>>> I don't see any violations.  We need to govern to policy here
>>>>>>>>>which is why
>>>>>>>>> Paul and Noreen are evaluating changes to the guidelines and our
>>>>>>>>> policies to make abuse more difficult.
>>>>>>>>> The second article is a competing vendor's reaction to the first.
>>>>>>>>> He makes some good points about the issues with Benchmark, but
>>>>>>>>>he also says
>>>>>>>>> that he hopes that it will be improved over time, and Dave has
>>>>>>>>>committed to
>>>>>>>>> that.  What I don't see is the vendor saying "...and Veracode
>>>>>>>>>has committed
>>>>>>>>> resources to help make the Benchmark more accurate across all
>>>>>>>>>tool sets".
>>>>>>>>> The Benchmark page is pretty clear that it does it's best to
>>>>>>>>>provide a
>>>>>>>>> benchmark without working exactly like a real-world application.
>>>>>>>>> Maybe some
>>>>>>>>> more disclaimer text about where the project is at today would
>>>>>>>>>be in order
>>>>>>>>> to validate some of Chris' concerns, but I hardly see this as
>>>>>>>>>"brand abuse"
>>>>>>>>> or a reason to demote the project.
>>>>>>>>> Please consider that I have spoken with both Dave and Jeff on
>>>>>>>>> topic and read much of the discussions around it before
>>>>>>>>>formulating my
>>>>>>>>> opinion.  I doubt that you have done the same so I'm not sure
>>>>>>>>>how you can
>>>>>>>>> claim that you have researched the issues and all parties
>>>>>>>>>involved when you
>>>>>>>>> haven't even spoken with the two people whom you are accusing of
>>>>>>>>> impropriety.  I have no bias here.  I am simply speaking with the
>>>>>>>>> individuals involved, looking at the currently OWASP policies and
>>>>>>>>> guidelines, and helping to determine our next steps.
>>>>>>>>> ~josh
>>>>>>>>> On Fri, Nov 27, 2015 at 12:22 PM, johanna curiel curiel
>>>>>>>>> <johanna.curiel at owasp.org> wrote:
>>>>>>>>>> >>While I agree with you that there has been some brand abuse,
>>>>>>>>>> >> was abuse by Contrast (specifically their marketing
>>>>>>>>>>department), and not by
>>>>>>>>>> >> "these gentlemen" as  you state.
>>>>>>>>>> Really? ..'some brand abuse'..this is more than brand abuse
>>>>>>>>>> Josh , please read also the article written by Jeff
>>>>>>>>>> And Veracode's reaction including others in Twitter
>>>>>>>>>> My strong advice is to research the issues and all the parties
>>>>>>>>>> involved before making statements
>>>>>>>>>> On Fri, Nov 27, 2015 at 2:07 PM, Josh Sokol
>>>>>>>>>><josh.sokol at owasp.org>
>>>>>>>>>> wrote:
>>>>>>>>>>> Jim,
>>>>>>>>>>> A concern was expressed to the Board and, frankly, I am
>>>>>>>>>>> by you saying that this was "brushed under the rug".  The
>>>>>>>>>>>Board delegated
>>>>>>>>>>> Matt to talk with Dave and they had a lengthy conversation on
>>>>>>>>>>>the subject.
>>>>>>>>>>> The Board delegated me to talk with Jeff and we had a lengthy
>>>>>>>>>>> on the subject.  If you do not trust in our abilities to read
>>>>>>>>>>>people, ask
>>>>>>>>>>> the right questions, and provide honest feedback about our
>>>>>>>>>>> then that's a bigger issue that we should take offline.  After
>>>>>>>>>>> conversations, we took the time to call a special two-hour
>>>>>>>>>>>session of the
>>>>>>>>>>> Board in order to discuss this subject (and only this
>>>>>>>>>>>subject).  We spoke
>>>>>>>>>>> about all facets of the issue at hand, about the challenges
>>>>>>>>>>>and possible
>>>>>>>>>>> solutions, and concluded on some very concrete next steps.
>>>>>>>>>>> While I agree with you that there has been some brand abuse, it
>>>>>>>>>>> was abuse by Contrast (specifically their marketing
>>>>>>>>>>>department), and not by
>>>>>>>>>>> "these gentlemen" as  you state.  Unless you can point to some
>>>>>>>>>>>sort of
>>>>>>>>>>> evidence showing that Jeff and/or Dave first-hand abused the
>>>>>>>>>>>brand, then I
>>>>>>>>>>> believe that you are speaking with your heart instead of with
>>>>>>>>>>>your head.  I
>>>>>>>>>>> appreciate your passion, but I label this as conspiracy theory
>>>>>>>>>>> without evidence to support your claims, I cannot accept it as
>>>>>>>>>>> other.
>>>>>>>>>>> ~josh
>>>>>>>>>>> On Fri, Nov 27, 2015 at 11:39 AM, Jim Manico
>>>>>>>>>>> <jim.manico at owasp.org> wrote:
>>>>>>>>>>>> Josh,
>>>>>>>>>>>> I stand by my comments and perspective, but I'm disheartened
>>>>>>>>>>>> you consider my presentation of facts (and the concerns of
>>>>>>>>>>>>many active
>>>>>>>>>>>> members of our community) as a "conspiracy theory".
>>>>>>>>>>>> In my experience, these kind of comments border on insults and
>>>>>>>>>>>> only cause folks to harden their opinions.
>>>>>>>>>>>> Once again I feel these gentlemen got away with a kind of
>>>>>>>>>>>> abuse that is very hurtful to the OWASP community but I am at
>>>>>>>>>>>>a loss as to
>>>>>>>>>>>> how handle or prevent these kinds of mishaps - especially
>>>>>>>>>>>>when board members
>>>>>>>>>>>> like yourself seem willing to - from what I see - brush it
>>>>>>>>>>>>under the rug.
>>>>>>>>>>>> --
>>>>>>>>>>>> Jim Manico
>>>>>>>>>>>> Global Board Member
>>>>>>>>>>>> OWASP Foundation
>>>>>>>>>>>> https://www.owasp.org
>>>>>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>>>>> On Nov 27, 2015, at 7:23 PM, Josh Sokol <josh.sokol at owasp.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Admittedly, this was my gut reaction at first as well.  I
>>>>>>>>>>>> linking all of these companies, people, and projects together
>>>>>>>>>>>>in my mind
>>>>>>>>>>>> (there are some loose links there) and painted a big
>>>>>>>>>>>>conspiracy picture
>>>>>>>>>>>> similar to what Jim and Dinis have stated.  But, after
>>>>>>>>>>>>speaking directly
>>>>>>>>>>>> with Jeff, and hearing about the conversation that Dave and
>>>>>>>>>>>>Matt had, I've
>>>>>>>>>>>> changed my mind.
>>>>>>>>>>>> I think it begins with the project itself.  If you aren't
>>>>>>>>>>>>sold on
>>>>>>>>>>>> the idea of the Benchmark, then you'll never be able to get
>>>>>>>>>>>>to the same
>>>>>>>>>>>> place.  My original line of thinking was that it was just a
>>>>>>>>>>>>bar for vendors
>>>>>>>>>>>> to compare their tools against eachother, but that's a bit
>>>>>>>>>>>>myopic.  We are
>>>>>>>>>>>> in an industry where things evolve very quickly.  As a
>>>>>>>>>>>>customer of these
>>>>>>>>>>>> tools, I know firsthand that something that a tool does today
>>>>>>>>>>>>may not be the
>>>>>>>>>>>> case a week from now.  Likewise, new features are being added
>>>>>>>>>>>>daily and I
>>>>>>>>>>>> need a point-in-time metric to be able to gauge continual
>>>>>>>>>>>> Cool, right?  But not a game changer.  The game changer part
>>>>>>>>>>>>comes when you
>>>>>>>>>>>> realize that by developing and evolving the tests that go
>>>>>>>>>>>>into the
>>>>>>>>>>>> Benchmark, we are moving the bar higher and higher.  We
>>>>>>>>>>>>(OWASP) are
>>>>>>>>>>>> effectively setting the standard by which these tools will be
>>>>>>>>>>>>compared.  A
>>>>>>>>>>>> tool that receives a lower score on the Benchmark today knows
>>>>>>>>>>>>exactly what
>>>>>>>>>>>> they need to work on in order to pass that test tomorrow and
>>>>>>>>>>>>we already have
>>>>>>>>>>>> examples of tools that have made improvements because of
>>>>>>>>>>>>their Benchmark
>>>>>>>>>>>> score (Ask Simon about ZAP's experience with the Benchmark).
>>>>>>>>>>>>I don't think
>>>>>>>>>>>> that anyone can argue that the Benchmark project isn't being
>>>>>>>>>>>>effective when
>>>>>>>>>>>> OWASP's own tools are being driven forward as a result of
>>>>>>>>>>>>using it.
>>>>>>>>>>>> But, but, but, Dave and Jeff own Aspect and have stock in
>>>>>>>>>>>> Contrast and Jeff is the Contrast CTO and Contrast got good
>>>>>>>>>>>>scores so it's a
>>>>>>>>>>>> conspiracy right?  Is there some code that allows Contrast to
>>>>>>>>>>>>use the
>>>>>>>>>>>> Benchmark?  Absolutely.  Can you really blame Dave for
>>>>>>>>>>>>starting his testing
>>>>>>>>>>>> on the effectiveness of the Benchmark with a tool that he
>>>>>>>>>>>>owned and is
>>>>>>>>>>>> familiar with?  If I were going to start a similar project,
>>>>>>>>>>>>there's no
>>>>>>>>>>>> question in my mind that I would begin my testing with the
>>>>>>>>>>>>tools that I have
>>>>>>>>>>>> available to me.  That said, is there code that allows other
>>>>>>>>>>>>tools to use
>>>>>>>>>>>> the Benchmark?  Absolutely.
>>>>>>>>>>>> Regarding "Dave has a history of breaching his duty to be
>>>>>>>>>>>> neutral", while I cannot comment on his past actions, I can
>>>>>>>>>>>>judge what we've
>>>>>>>>>>>> seen recently.  Matt saw a presentation from Dave on the
>>>>>>>>>>>>Benchmark at a
>>>>>>>>>>>> conference in Chicago.  He said that he felt that the message
>>>>>>>>>>>> appropriate and while IAST tools were mentioned as receiving
>>>>>>>>>>>>higher scores,
>>>>>>>>>>>> it wasn't a "Contrast is the best" type of message, more of a
>>>>>>>>>>>>generality.  I
>>>>>>>>>>>> saw a very similar (if not the same) talk by Jeff at LASCON
>>>>>>>>>>>>2015 and the
>>>>>>>>>>>> message was exactly the same.  I watched the talk expecting
>>>>>>>>>>>>some sort of
>>>>>>>>>>>> impropriety, but found none.  So, perhaps Dave has abused
>>>>>>>>>>>>some privilege
>>>>>>>>>>>> granted to him in the past, but what I've seen from him at
>>>>>>>>>>>>this point, with
>>>>>>>>>>>> respect to the Benchmark, has been appropriate.
>>>>>>>>>>>> You have a very good point with respect to the Contrast
>>>>>>>>>>>> message around the Benchmark.  It's been completely absurd,
>>>>>>>>>>>>over the top,
>>>>>>>>>>>> and, in my personal opinion, intolerable.  In fact, I
>>>>>>>>>>>>experienced the same
>>>>>>>>>>>> thing that you talked about with them at LASCON 2015 where
>>>>>>>>>>>>they stood in
>>>>>>>>>>>> front of the door of the room Jeff was speaking in and
>>>>>>>>>>>>scanned attendees as
>>>>>>>>>>>> they went into the talk.  I agree that these types of
>>>>>>>>>>>>aggressive marketing
>>>>>>>>>>>> tactics cannot be tolerated at OWASP.  In addition, we have
>>>>>>>>>>>>seen several
>>>>>>>>>>>> marketing messages from them effectively implying that OWASP
>>>>>>>>>>>> Contrast.  Clearly this is not OK.  I've spoken with Jeff
>>>>>>>>>>>>about it and we
>>>>>>>>>>>> agreed that it is not in the Benchmark's best interest to
>>>>>>>>>>>>have this
>>>>>>>>>>>> aggressive Contrast marketing around it at such an early
>>>>>>>>>>>>stage.  He has said
>>>>>>>>>>>> that he is not responsible for Contrast's marketing team, but
>>>>>>>>>>>>that he would
>>>>>>>>>>>> speak with the people who are.  I haven't seen a single
>>>>>>>>>>>>message from them
>>>>>>>>>>>> since so I'm guessing that he's made good on this promise.
>>>>>>>>>>>>While that's an
>>>>>>>>>>>> excellent start, OWASP's takeaway here should be that we need
>>>>>>>>>>>>to do a better
>>>>>>>>>>>> job with our brand usage guidelines both in terms of the
>>>>>>>>>>>>wording and
>>>>>>>>>>>> enforcement.  There are many other companies out there that
>>>>>>>>>>>>use the OWASP
>>>>>>>>>>>> brand and I think that we agree that selective enforcement
>>>>>>>>>>>>against Contrast
>>>>>>>>>>>> is not the right answer.  Paul and Noreen are actively
>>>>>>>>>>>>working on this.
>>>>>>>>>>>> Either way, I think that implying that activities from a
>>>>>>>>>>>>vendor's marketing
>>>>>>>>>>>> department means that the project is not objective is not
>>>>>>>>>>>>inappropriate.  If
>>>>>>>>>>>> we feel that the project is not objective, then separate
>>>>>>>>>>>>measures need to be
>>>>>>>>>>>> taken to drive contribution diversity into it.  That I
>>>>>>>>>>>>absolutely agree with
>>>>>>>>>>>> and the message from Dave was that he would love to have more
>>>>>>>>>>>> to his project.  But, seeing as we cannot force people to
>>>>>>>>>>>>work on it, this
>>>>>>>>>>>> becomes a matter of "put up or shut up".  The same goes for
>>>>>>>>>>>>the experts that
>>>>>>>>>>>> you said reviewed the code.  If they feel that it is somehow
>>>>>>>>>>>>skewed towards
>>>>>>>>>>>> Contrast, they have the power to change that.  Now, if
>>>>>>>>>>>>someone tries to
>>>>>>>>>>>> participate and Dave tells them "No thanks", then I agree we
>>>>>>>>>>>>have a problem,
>>>>>>>>>>>> but I don't hear anyone inferring that happened.
>>>>>>>>>>>> Please, let's drop the conspiracy theories and focus on the
>>>>>>>>>>>> tangible things that we can do to help an OWASP project to be
>>>>>>>>>>>> successful.  Help find more participants to drive diversity,
>>>>>>>>>>>>update our
>>>>>>>>>>>> brand usage guidelines to prevent abuse, enforce them widely,
>>>>>>>>>>>>etc.  Thank
>>>>>>>>>>>> you.
>>>>>>>>>>>> ~josh
>>>>>>>>>>>> On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico
>>>>>>>>>>>> <jim.manico at owasp.org> wrote:
>>>>>>>>>>>>> Dinis,
>>>>>>>>>>>>> Like a rare celestial moment when all the planets plus Pluto
>>>>>>>>>>>>> aligned, I just read your email on the future of OWASP
>>>>>>>>>>>>>projects thinking,
>>>>>>>>>>>>> "Dinis is spot on".
>>>>>>>>>>>>> Reflecting on projects I manage or work on...
>>>>>>>>>>>>> The Java Encoder and HTML Sanitizer are likely best moved to
>>>>>>>>>>>>> Apache now that they have reached a measure of adoption and
>>>>>>>>>>>>>maturity. Apache
>>>>>>>>>>>>> would be a much better long term custodian. Perhaps the same
>>>>>>>>>>>>>for AppSensor,
>>>>>>>>>>>>> but not my project - just thinking out loud.
>>>>>>>>>>>>> Other similar defensive projects are still being noodled on,
>>>>>>>>>>>>> OWASP is a decent home for these research efforts.
>>>>>>>>>>>>> The whole tools category is also something to consider.
>>>>>>>>>>>>> Dependency Check and of course ZAP are some of the best
>>>>>>>>>>>>>projects that OWASP
>>>>>>>>>>>>> offers, are they best served where they are today? Both have
>>>>>>>>>>>>> communities of developers but I don't see the foundation
>>>>>>>>>>>>>doing much to
>>>>>>>>>>>>> support these efforts.
>>>>>>>>>>>>> ASVS has the opportunity to effect massive change, I would to
>>>>>>>>>>>>> love to see major investment and volunteer activity here.
>>>>>>>>>>>>>Pro tech writer,
>>>>>>>>>>>>> detailed discourses on each individual requirement, etc. If
>>>>>>>>>>>>>I was king (and
>>>>>>>>>>>>> I am not, at all) I would invest in ASVS on a 6 figure
>>>>>>>>>>>>>scale. (And who
>>>>>>>>>>>>> started ASVS? Jeff, Dave and Boberski, hat tip to such a
>>>>>>>>>>>>>marvelous idea). Or
>>>>>>>>>>>>> maybe moving ASVS to the W3C or IETF would help it grow?
>>>>>>>>>>>>> The Proactive Controls was a pet project but as we approach
>>>>>>>>>>>>> we have several active/awesome volunteers working on it. We
>>>>>>>>>>>>>will be making
>>>>>>>>>>>>> the doc "world editable" to make contributions easy. OWASP 
>>>>>>>>>>>>>seems like a good
>>>>>>>>>>>>> home for such an awareness doc. Same with T10, especially if 
>>>>>>>>>>>>>community edits
>>>>>>>>>>>>> are welcome.
>>>>>>>>>>>>> Anyhow, I'm with you on this Dinis. Once a project starts to
>>>>>>>>>>>>> reach production quality, spinning off the project as an 
>>>>>>>>>>>>>external project or
>>>>>>>>>>>>> moving it to a different foundation where managing 
>>>>>>>>>>>>>production software or
>>>>>>>>>>>>> formal standards is their thing seems realistic.
>>>>>>>>>>>>> I don't have all the answers here, but your email certainly
>>>>>>>>>>>>> resonated with me.
>>>>>>>>>>>>> Aloha,
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Jim Manico
>>>>>>>>>>>>> Global Board Member
>>>>>>>>>>>>> OWASP Foundation
>>>>>>>>>>>>> https://www.owasp.org
>>>>>>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>>>>>> On Nov 26, 2015, at 11:26 PM, Dinis Cruz 
>>>>>>>>>>>>><dinis.cruz at owasp.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Jim's reading of this situation is exactly my view on the 
>>>>>>>>>>>>> of the Contrast tool and how it has been 'pushing' the rules 
>>>>>>>>>>>>>of engagement
>>>>>>>>>>>>> to an very 'fuzzy' moral/ethical/commercial limit :)
>>>>>>>>>>>>> As per my last email, a key problem here is the 'perceived
>>>>>>>>>>>>> expectation' of what is an OWASP project, and how it should 
>>>>>>>>>>>>>be consumed.
>>>>>>>>>>>>> If you look at the OWASP benchmark as a research project, 
>>>>>>>>>>>>> the only way it could be making the kind of claims it makes 
>>>>>>>>>>>>>(and have
>>>>>>>>>>>>> credibility) is if it had evolved from OWASP, with its own 
>>>>>>>>>>>>> community
>>>>>>>>>>>>> On 26 November 2015 at 21:01, Jim Manico 
>>>>>>>>>>>>><jim.manico at owasp.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I have a different take on this situation but my opinion is 
>>>>>>>>>>>>>> "minority opinion". I will respect the rest of the boards 
>>>>>>>>>>>>>>take on this, but
>>>>>>>>>>>>>> here is how I see it.
>>>>>>>>>>>>>> First of all, Jeff has stated that he feels I am attacking 
>>>>>>>>>>>>>> personally from a past personal grudge, and frankly I do 
>>>>>>>>>>>>>>not fault him for
>>>>>>>>>>>>>> that perspective since we definitely have history with 
>>>>>>>>>>>>>>conflict. So it's
>>>>>>>>>>>>>> fair to take my opinion on this with a grain of salt.
>>>>>>>>>>>>>> I look at this situation from the perspective of a forensic
>>>>>>>>>>>>>> investigator.
>>>>>>>>>>>>>> 1) The Benchmark project had Contrast hooks and only 
>>>>>>>>>>>>>> hooks in it when I reviewed it so this leads me to believe 
>>>>>>>>>>>>>>that the project
>>>>>>>>>>>>>> was clearly built with Contrast in mind from the ground up, 
>>>>>>>>>>>>>>at least in some
>>>>>>>>>>>>>> way.
>>>>>>>>>>>>>> 3) Dave has a history of breaching his duty to be vendor
>>>>>>>>>>>>>> neutral. He was gifted with a keynote in South Korea a few 
>>>>>>>>>>>>>>years ago, and
>>>>>>>>>>>>>> used that opportunity to discuss and pitch Contrast, on 
>>>>>>>>>>>>>>stage, during a
>>>>>>>>>>>>>> keynote - with Contrast specific slides. This is just 
>>>>>>>>>>>>>>supporting evidence of
>>>>>>>>>>>>>> his intention at OWASP to push Contrast in ways that I 
>>>>>>>>>>>>>>think are against the
>>>>>>>>>>>>>> intentions and goals of our foundation.
>>>>>>>>>>>>>> 3) Other experts have reviewed the project and felt that 
>>>>>>>>>>>>>> of the tests were very slanted and almost contrived to 
>>>>>>>>>>>>>>support Contrast. I
>>>>>>>>>>>>>> can drag those folks into this conversation, but I do not 
>>>>>>>>>>>>>>think that would
>>>>>>>>>>>>>> help in any way. So it's fair to call this point heresy.
>>>>>>>>>>>>>> 4) I do not see this project as revolutionary, at all. Every
>>>>>>>>>>>>>> vendor has their own test suite tuned for their tool. As 
>>>>>>>>>>>>>>the benchmark
>>>>>>>>>>>>>> stands today, I see it as just another vendors 
>>>>>>>>>>>>>>product-specific benchmark.
>>>>>>>>>>>>>> Mass collaboration from many vendors is not just a "nice to 
>>>>>>>>>>>>>>have" but a base
>>>>>>>>>>>>>> requirement to get even close to useful for objective tool 
>>>>>>>>>>>>>> 5) Jeff stating that his Marketing people went over the 
>>>>>>>>>>>>>>line is
>>>>>>>>>>>>>> also an admission that - well, they went over the line. By 
>>>>>>>>>>>>>>the same token
>>>>>>>>>>>>>> Jeff was in his booth at AppSec USA surrounded by benchmark 
>>>>>>>>>>>>>> material, discussing this to prospects and he even asked me 
>>>>>>>>>>>>>>and Mr Coates to
>>>>>>>>>>>>>> wade into this debate and support Dave. So to say he was 
>>>>>>>>>>>>>>not involved and it
>>>>>>>>>>>>>> was only his marketing people seems a stretch at best.
>>>>>>>>>>>>>> 6) The Contrast marketing team was wandering around the
>>>>>>>>>>>>>> conference zapping folks to get leads, and I asked them to 
>>>>>>>>>>>>>>stay in their
>>>>>>>>>>>>>> booth, which is standard conference policy. These folks 
>>>>>>>>>>>>>>know better but are
>>>>>>>>>>>>>> again going over the line to sell product at OWASP. There 
>>>>>>>>>>>>>>is a better way
>>>>>>>>>>>>>> (like focusing on product capability and language support, 
>>>>>>>>>>>>>>have consistent +
>>>>>>>>>>>>>> stellar customer service, have a humble and gracious 
>>>>>>>>>>>>>>attitude to all
>>>>>>>>>>>>>> prospects and customers, actively participate in OWASP in a 
>>>>>>>>>>>>>>vendor neutral
>>>>>>>>>>>>>> and community supportive way, etc).
>>>>>>>>>>>>>> Please note, I think Contrast is a decent tool, I've 
>>>>>>>>>>>>>>offered to
>>>>>>>>>>>>>> resell in the past, and I have recommended it in certain 
>>>>>>>>>>>>>>situations - even
>>>>>>>>>>>>>> after this situation arose. I'm stating this out of 
>>>>>>>>>>>>>>honestly and desire to
>>>>>>>>>>>>>> put my cards on the table. I truly want Jeff and Dave to be 
>>>>>>>>>>>>>>successful. They
>>>>>>>>>>>>>> have dedicated their lives to AppSec and if anyone should 
>>>>>>>>>>>>>>win big-time, I
>>>>>>>>>>>>>> hope it's them. I even told Jeff I hope he hits the mother 
>>>>>>>>>>>>>>load and donates
>>>>>>>>>>>>>> a little back to OWASP.
>>>>>>>>>>>>>> However, my instinct and evidence tell me that they both 
>>>>>>>>>>>>>> over the line in the use of the OWASP brand to sell product.
>>>>>>>>>>>>>> Now, Jeff makes a good point. We as a board and staff are 
>>>>>>>>>>>>>> poor at enforcing brand management policy and it's not fair 
>>>>>>>>>>>>>>to single out
>>>>>>>>>>>>>> Contrast, when many other vendors violate the brand, IMO. 
>>>>>>>>>>>>>>Just google OWASP
>>>>>>>>>>>>>> and watch the ads fly that use the OWASP name to sell 
>>>>>>>>>>>>>> Also, any and every request that was made of Dave to adjust 
>>>>>>>>>>>>>> project for the sake of vendor neutrality was taken very 
>>>>>>>>>>>>>> Regardless of Daves past intentions, he is clearly trying 
>>>>>>>>>>>>>>to do the right
>>>>>>>>>>>>>> thing moving forward.
>>>>>>>>>>>>>> I look to "postels principle" in this situation (this is
>>>>>>>>>>>>>> otherwise known as the "robustness principle" and dates 
>>>>>>>>>>>>>>back to the creation
>>>>>>>>>>>>>> of TCP) . This is paraphrased as, "Be liberal in what you 
>>>>>>>>>>>>>>take from others
>>>>>>>>>>>>>> but be conservative in what you dish out". So I think it's 
>>>>>>>>>>>>>>critical that
>>>>>>>>>>>>>> OWASP and any OWASP resource present itself in a strict 
>>>>>>>>>>>>>>vendor neutral way.
>>>>>>>>>>>>>> But unless OWASP wants to be much more "even" in the 
>>>>>>>>>>>>>>enforcement of brand
>>>>>>>>>>>>>> policy across the board to all violators, we should be 
>>>>>>>>>>>>>>fairly lax in the
>>>>>>>>>>>>>> enforcement of these issues from the outside world.
>>>>>>>>>>>>>> I am trying to be objective here. My trigonometry teacher 
>>>>>>>>>>>>>> told me "I'd fail my mother" when I asked him if he would 
>>>>>>>>>>>>>>ever fail me (I
>>>>>>>>>>>>>> was an A student). If my mother owned a security company 
>>>>>>>>>>>>>>and tried the same
>>>>>>>>>>>>>> stunt, I'd have the same opinions about her actions as well.
>>>>>>>>>>>>>> So what next? Well hello from the other side. I'm going 
>>>>>>>>>>>>>>back to
>>>>>>>>>>>>>> listening to Adele's new album where I can sit in my deep 
>>>>>>>>>>>>>>feelings and
>>>>>>>>>>>>>> reflect upon what the OWASP foundation has done to enrich 
>>>>>>>>>>>>>>my life. I would
>>>>>>>>>>>>>> much rather keep out of this (and any other conflict laden 
>>>>>>>>>>>>>>situation at
>>>>>>>>>>>>>> OWASP), but I feel it's my responsibility to speak up.
>>>>>>>>>>>>>> Aloha,
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Jim Manico
>>>>>>>>>>>>>> Global Board Member
>>>>>>>>>>>>>> OWASP Foundation
>>>>>>>>>>>>>> https://www.owasp.org
>>>>>>>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>>>>>>> On Nov 26, 2015, at 9:09 PM, Josh Sokol 
>>>>>>>>>>>>>><josh.sokol at owasp.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I would be happy to provide an update.
>>>>>>>>>>>>>> Matt Konda and Dave Wichers, the Benchmark Project Leader, 
>>>>>>>>>>>>>> a conversation a few weeks back.  To summarize their 
>>>>>>>>>>>>>>conversation, Dave
>>>>>>>>>>>>>> acknowledges the currently lack of diversity in his project 
>>>>>>>>>>>>>>and it is his
>>>>>>>>>>>>>> sincere desire to drive more people to it to help.  He also 
>>>>>>>>>>>>>>acknowledges the
>>>>>>>>>>>>>> issues with Contrast's extreme marketing around the project 
>>>>>>>>>>>>>>and feels that
>>>>>>>>>>>>>> it is in everyone's best interests for them to curb it 
>>>>>>>>>>>>>>back.  While he does
>>>>>>>>>>>>>> have an ownership stake in Contrast, he works at Aspect and 
>>>>>>>>>>>>>>has no control
>>>>>>>>>>>>>> over the marketing messages that they are putting out 
>>>>>>>>>>>>>>there.  From the Board
>>>>>>>>>>>>>> perspective, there has been no evidence of any impropriety 
>>>>>>>>>>>>>>on Dave's part
>>>>>>>>>>>>>> and it should be our goal to drive more diversity into the 
>>>>>>>>>>>>>>project to
>>>>>>>>>>>>>> support Dave.  Dave appears to be sincere in his desires to 
>>>>>>>>>>>>>>create a tool
>>>>>>>>>>>>>> where OWASP can tell vendors what we expect from their 
>>>>>>>>>>>>>>tools.  If the main
>>>>>>>>>>>>>> issue is that only members of Aspect are working on it, 
>>>>>>>>>>>>>>then the best thing
>>>>>>>>>>>>>> that we can do is try to get him some outside assistance.  
>>>>>>>>>>>>>>We are also
>>>>>>>>>>>>>> asking that the project be opened up to commits via Git so 
>>>>>>>>>>>>>>that outsiders
>>>>>>>>>>>>>> can push commits to it.
>>>>>>>>>>>>>> Josh Sokol and Jeff Williams, the CTO of Contrast, had a
>>>>>>>>>>>>>> conversation a few weeks back.  To summarize their 
>>>>>>>>>>>>>>conversation, Jeff
>>>>>>>>>>>>>> believes that the work that Dave is doing on the Benchmark 
>>>>>>>>>>>>>>is a game changer
>>>>>>>>>>>>>> in that it gives OWASP the power in dictating what these 
>>>>>>>>>>>>>>tools need to be
>>>>>>>>>>>>>> finding.  He wants the Benchmark to be successful and 
>>>>>>>>>>>>>>understands that it
>>>>>>>>>>>>>> needs to be diverse in order to be trusted.  He recognizes 
>>>>>>>>>>>>>>that Dave is
>>>>>>>>>>>>>> trying to do that and does not want the marketing message 
>>>>>>>>>>>>>>from Contrast to
>>>>>>>>>>>>>> interfere with his efforts.  Jeff felt that the "Lab" 
>>>>>>>>>>>>>>status granted to
>>>>>>>>>>>>>> Benchmark meant that it was ready for mainstream adoption, 
>>>>>>>>>>>>>>that it had 21k
>>>>>>>>>>>>>> tests, and was almost a year old, and didn't see anything 
>>>>>>>>>>>>>>wrong with
>>>>>>>>>>>>>> marketing their results, but has agreed to talk to their 
>>>>>>>>>>>>>>marketing team to
>>>>>>>>>>>>>> get them to lay off that message for now.  From the Board 
>>>>>>>>>>>>>>perspective, we
>>>>>>>>>>>>>> have come to the realization that our brand usage 
>>>>>>>>>>>>>>guidelines need an
>>>>>>>>>>>>>> overhaul to clarify what is and is not allowed.  We have 
>>>>>>>>>>>>>>made a few
>>>>>>>>>>>>>> proposals and have reached out to Mozilla to gain more 
>>>>>>>>>>>>>>insight on their
>>>>>>>>>>>>>> guidelines and even ask for assistance.  Noreen and Paul 
>>>>>>>>>>>>>>are taking lead on
>>>>>>>>>>>>>> these efforts.
>>>>>>>>>>>>>> There is a note in the notes that the Board was supposed to
>>>>>>>>>>>>>> follow up with an open letter to the community and 
>>>>>>>>>>>>>>companies involved
>>>>>>>>>>>>>> describing our review and actions.  I don't think that has 
>>>>>>>>>>>>>>happened so I
>>>>>>>>>>>>>> will remind the person who took on that action item.
>>>>>>>>>>>>>> I'm happy to answer any questions that you may have.
>>>>>>>>>>>>>> ~josh
>>>>>>>>>>>>>> On Thu, Nov 26, 2015 at 11:55 AM, Tobias
>>>>>>>>>>>>>> <tobias.gondrom at owasp.org> wrote:
>>>>>>>>>>>>>>> There have been several conversations on that matter and a
>>>>>>>>>>>>>>> dedicated call. Unfortunately for personal reasons I could 
>>>>>>>>>>>>>>>not attend the
>>>>>>>>>>>>>>> last call as it was at 04:00am my local time, but all 
>>>>>>>>>>>>>>>other board members
>>>>>>>>>>>>>>> did participate.
>>>>>>>>>>>>>>> Could please one of my fellow board members give an update.
>>>>>>>>>>>>>>> Best, Tobias
>>>>>>>>>>>>>>> On 26/11/15 18:04, Timo Goosen wrote:
>>>>>>>>>>>>>>> I would also like to know the answer to Simon's question. 
>>>>>>>>>>>>>>> need to get rid of bad apples in OWASP in my opinion, 
>>>>>>>>>>>>>>>there are too many
>>>>>>>>>>>>>>> people just using the OWASP "name" or "brand" to improve 
>>>>>>>>>>>>>>>their own financial
>>>>>>>>>>>>>>> situation or career.
>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>> Timo
>>>>>>>>>>>>>>> On Thu, Nov 26, 2015 at 1:13 PM, psiinon 
>>>>>>>>>>>>>>><psiinon at gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> Paul, and the rest of the board,
>>>>>>>>>>>>>>>> Its been over 2 months since I raised this issue.
>>>>>>>>>>>>>>>> Whats happening?
>>>>>>>>>>>>>>>> Has the board even discussed it?
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>> On Tue, Oct 20, 2015 at 10:00 PM, Paul Ritchie
>>>>>>>>>>>>>>>> <paul.ritchie at owasp.org> wrote:
>>>>>>>>>>>>>>>>> Eoin, Johanna, All:
>>>>>>>>>>>>>>>>> In an earlier email, Josh Sokol mentioned that he will be
>>>>>>>>>>>>>>>>> speaking in the next day or 2 to their CTO, while at 
>>>>>>>>>>>>>>>>>LASCON, as a
>>>>>>>>>>>>>>>>> representative of the OWASP Board.  Following that 
>>>>>>>>>>>>>>>>>feedback, the Board has
>>>>>>>>>>>>>>>>> action to take the next steps.
>>>>>>>>>>>>>>>>> Just an FYI that all comments are recognized and action 
>>>>>>>>>>>>>>>>> being taken.
>>>>>>>>>>>>>>>>> Paul
>>>>>>>>>>>>>>>>> Best Regards, Paul Ritchie
>>>>>>>>>>>>>>>>> OWASP Executive Director
>>>>>>>>>>>>>>>>> paul.ritchie at owasp.org
>>>>>>>>>>>>>>>>> On Tue, Oct 20, 2015 at 1:54 PM, johanna curiel curiel
>>>>>>>>>>>>>>>>> <johanna.curiel at owasp.org> wrote:
>>>>>>>>>>>>>>>>>> Time for owasp to do a public statement and put a clear
>>>>>>>>>>>>>>>>>> story regarding this abusive behavior of Owasp brand
>>>>>>>>>>>>>>>>>> On Tuesday, October 20, 2015, Eoin Keary
>>>>>>>>>>>>>>>>>> <eoin.keary at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>> Folks,
>>>>>>>>>>>>>>>>>>> The project should be immediately shelved it's simply 
>>>>>>>>>>>>>>>>>>> form.
>>>>>>>>>>>>>>>>>>> This is damaging to OWASP, the industry and exactly 
>>>>>>>>>>>>>>>>>>> OWASP is not about.
>>>>>>>>>>>>>>>>>>> There is a clear conflict of interest and distinct 
>>>>>>>>>>>>>>>>>>>lack of
>>>>>>>>>>>>>>>>>>> science behind the claims made by Contrast.
>>>>>>>>>>>>>>>>>>> Eoin Keary
>>>>>>>>>>>>>>>>>>> OWASP Volunteer
>>>>>>>>>>>>>>>>>>> @eoinkeary
>>>>>>>>>>>>>>>>>>> On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel
>>>>>>>>>>>>>>>>>>> <johanna.curiel at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>> At the moment we did the project review, we observed 
>>>>>>>>>>>>>>>>>>> the project did not have enough testing to be 
>>>>>>>>>>>>>>>>>>>considered in any form as
>>>>>>>>>>>>>>>>>>> 'ready'  for benchmarking, neither that it had yet the 
>>>>>>>>>>>>>>>>>>>community adoption,
>>>>>>>>>>>>>>>>>>> however technically speaking as it has been classified 
>>>>>>>>>>>>>>>>>>>by the leaders, the
>>>>>>>>>>>>>>>>>>> project is at the beta stage.
>>>>>>>>>>>>>>>>>>> Indeed , Dave had the push to have the project reviewed
>>>>>>>>>>>>>>>>>>> but it was never clear that later on the project was 
>>>>>>>>>>>>>>>>>>>going to be advertisied
>>>>>>>>>>>>>>>>>>> this way. That all happend after the presentation at 
>>>>>>>>>>>>>>>>>>> I had my concerns regarding how sensitive is the 
>>>>>>>>>>>>>>>>>>> of the project ,but I think we should allow project 
>>>>>>>>>>>>>>>>>>>leaders to develop their
>>>>>>>>>>>>>>>>>>> communication strategy even if this has conflict of 
>>>>>>>>>>>>>>>>>>>interest. It all depends
>>>>>>>>>>>>>>>>>>> how they behave and how they manage this.
>>>>>>>>>>>>>>>>>>> On Tuesday, October 6, 2015, Michael Coates
>>>>>>>>>>>>>>>>>>> <michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>>> It's not really that formal to add to the agenda, 
>>>>>>>>>>>>>>>>>>>>just a
>>>>>>>>>>>>>>>>>>>> wiki that we add in the text.
>>>>>>>>>>>>>>>>>>>> I think you can safely assume it will get the 
>>>>>>>>>>>>>>>>>>>> discussion.
>>>>>>>>>>>>>>>>>>>> On Oct 6, 2015, at 7:16 AM, psiinon 
>>>>>>>>>>>>>>>>>>>><psiinon at gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Really?? Its not on the agenda yet for the next 
>>>>>>>>>>>>>>>>>>>> How does it get added to the agenda?
>>>>>>>>>>>>>>>>>>>> And that was a formal request if that makes any
>>>>>>>>>>>>>>>>>>>> difference :)
>>>>>>>>>>>>>>>>>>>> I'm all in favour of getting the facts straight before
>>>>>>>>>>>>>>>>>>>> any actions are taken, hence my request for an 
>>>>>>>>>>>>>>>>>>>>'ethical review' or whatever
>>>>>>>>>>>>>>>>>>>> it should be called.
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates
>>>>>>>>>>>>>>>>>>>> <michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>>>> First step is to get all of our information straight so
>>>>>>>>>>>>>>>>>>>>> we're clear on where things are at.
>>>>>>>>>>>>>>>>>>>>> This was not on the board agenda last meeting and is
>>>>>>>>>>>>>>>>>>>>> also not on the next agenda as of yet (of course it could 
always be added if
>>>>>>>>>>>>>>>>>>>>> needed).
>>>>>>>>>>>>>>>>>>>>> We are aware that people have raised questions though.
>>>>>>>>>>>>>>>>>>>>> I'm hoping we can get a clear understanding of all the 
facts and then
>>>>>>>>>>>>>>>>>>>>> discuss if changes are needed.
>>>>>>>>>>>>>>>>>>>>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> Hey Michael,
>>>>>>>>>>>>>>>>>>>>> Is the board going to take any action?
>>>>>>>>>>>>>>>>>>>>> Were there any discussions about this controversy in the
>>>>>>>>>>>>>>>>>>>>> board meeting at AppSec USA?
>>>>>>>>>>>>>>>>>>>>> If not will it be on the agenda for the meeting on
>>>>>>>>>>>>>>>>>>>>> October 14th?
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates
>>>>>>>>>>>>>>>>>>>>> <michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>>>>>>> I posted the below message earlier today. At this point
>>>>>>>>>>>>>>>>>>>>>> my goal is to just gain clarity over the current reality 
and ideally drive
>>>>>>>>>>>>>>>>>>>>>> to a shared state of success. This message doesn't seem 
to be reflected in
>>>>>>>>>>>>>>>>>>>>>> the list yet. It could be because my membership hasn't 
been approved or
>>>>>>>>>>>>>>>>>>>>>> because of mail list delays (I miss Google groups). But I 
think these
>>>>>>>>>>>>>>>>>>>>>> questions will start the conversation.
>>>>>>>>>>>>>>>>>>>>>> (This was just me asking questions as a curious Owasp
>>>>>>>>>>>>>>>>>>>>>> member, not any action on behalf of the board)
>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>>>>>>>>>>>> From: Michael Coates <michael.coates at owasp.org>
>>>>>>>>>>>>>>>>>>>>>> Date: October 5, 2015 at 6:20:23 PM PDT
>>>>>>>>>>>>>>>>>>>>>> To: owasp-benchmark-project at lists.owasp.org
>>>>>>>>>>>>>>>>>>>>>> Subject: Project Questions
>>>>>>>>>>>>>>>>>>>>>> OWASP Benchmark List,
>>>>>>>>>>>>>>>>>>>>>> I've heard more about this project and am excited about
>>>>>>>>>>>>>>>>>>>>>> the idea of an independent perspective of tool 
performance. I'm trying to
>>>>>>>>>>>>>>>>>>>>>> understand a few things to better respond to questions 
from those in the
>>>>>>>>>>>>>>>>>>>>>> security & OWASP community.
>>>>>>>>>>>>>>>>>>>>>> In my mind there are two big areas for consideration in
>>>>>>>>>>>>>>>>>>>>>> a benchmark process.
>>>>>>>>>>>>>>>>>>>>>> 1. Are the benchmarks testing the right areas?
>>>>>>>>>>>>>>>>>>>>>> 2. Is the process for creating the benchmark objective
>>>>>>>>>>>>>>>>>>>>>> & free from conflicts of interest.
>>>>>>>>>>>>>>>>>>>>>> I think as a group OWASP is the right body to align on
>>>>>>>>>>>>>>>>>>>>>> #1.
>>>>>>>>>>>>>>>>>>>>>> I'd like to ask for some clarifications on item #2. I
>>>>>>>>>>>>>>>>>>>>>> think it's important to avoid actual conflict of interest 
and also the
>>>>>>>>>>>>>>>>>>>>>> appearance of conflict of interest. The former is obvious 
why we mustn't
>>>>>>>>>>>>>>>>>>>>>> have that, the latter is critical so others have faith in 
the tool, process
>>>>>>>>>>>>>>>>>>>>>> and outputs of the process when viewing or hearing about 
the project.
>>>>>>>>>>>>>>>>>>>>>> 1) Can we clarify whether other individuals have
>>>>>>>>>>>>>>>>>>>>>> submitted meaningful code to the project?
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> Nearly all the code commits have come from 1 person
>>>>>>>>>>>>>>>>>>>>>> (project lead).
>>>>>>>>>>>>>>>>>>>>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>>>>>>>>>>>>>>>>>>>>> 2) Can we clarify the contributions of others and their
>>>>>>>>>>>>>>>>>>>>>> represented organizations?
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> The acknowledgements tab listed two developers (Juan
>>>>>>>>>>>>>>>>>>>>>> Gama & Nick Sanidas) both who work at the same company as 
the project lead.
>>>>>>>>>>>>>>>>>>>>>> It seems other people have submitted some small amounts 
of material, but
>>>>>>>>>>>>>>>>>>>>>> overall it seems all development has come from the same 
>>>>>>>>>>>>>>>>>>>>>> 3) Can we clarify in what ways we've mitigated the
>>>>>>>>>>>>>>>>>>>>>> potential conflict of interest and also the appearance of 
a conflict of
>>>>>>>>>>>>>>>>>>>>>> interest? This seems like the largest blocker for wide 
spread acceptance of
>>>>>>>>>>>>>>>>>>>>>> this project and the biggest risk.
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> The project lead and both of the project developers
>>>>>>>>>>>>>>>>>>>>>> works for a company with very close ties to one of the 
companies that is
>>>>>>>>>>>>>>>>>>>>>> evaluated by this project. Further, it appears the 
company is performing
>>>>>>>>>>>>>>>>>>>>>> very well on the project tests.
>>>>>>>>>>>>>>>>>>>>>> 4) If we are going to list tool vendors then I'd
>>>>>>>>>>>>>>>>>>>>>> recommend listing multiple vendors for each category.
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> The tools page only lists 1 IAST tool. Since this is
>>>>>>>>>>>>>>>>>>>>>> the point of the potential conflict of interest it is 
important to list
>>>>>>>>>>>>>>>>>>>>>> numerous IAST tools.
>>>>>>>>>>>>>>>>>>>>>> 5) Diverse body with multiple points of view
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> There is no indication that multiple stakeholders are
>>>>>>>>>>>>>>>>>>>>>> present to review and decide on the future of this 
project. If they exist, a
>>>>>>>>>>>>>>>>>>>>>> new section should be added to the project page to raise 
awareness. If they
>>>>>>>>>>>>>>>>>>>>>> don't exist, we should reevaluate how we are obtaining an 
independent view
>>>>>>>>>>>>>>>>>>>>>> of the testing process.
>>>>>>>>>>>>>>>>>>>>>> Again, I think the idea of the project is great. From
>>>>>>>>>>>>>>>>>>>>>> my perspective clarifying these questions will help 
ensure the project is
>>>>>>>>>>>>>>>>>>>>>> not only objective, but also perceived as objective from 
someone reviewing
>>>>>>>>>>>>>>>>>>>>>> the material. Ultimately this will contribute to the 
success and growth of
>>>>>>>>>>>>>>>>>>>>>> the project.
>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Michael Coates
>>>>>>>>>>>>>>>>>>>>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> OK, based on the concerns raised so far I think the
>>>>>>>>>>>>>>>>>>>>>> board should initiate a review of the OWASP Benchmark 
>>>>>>>>>>>>>>>>>>>>>> I'm not raising a formal complaint against it, I'm just
>>>>>>>>>>>>>>>>>>>>>> requesting a review.
>>>>>>>>>>>>>>>>>>>>>> And I dont think it needs a 'standard' project review -
>>>>>>>>>>>>>>>>>>>>>> Johanna has already done a very good job of this.
>>>>>>>>>>>>>>>>>>>>>> Not sure what sort of review you'd call it, I'll leave
>>>>>>>>>>>>>>>>>>>>>> the naming to others :)
>>>>>>>>>>>>>>>>>>>>>> I'm concerned that we have an OWASP project lead by a
>>>>>>>>>>>>>>>>>>>>>> company who has a clear commercial stake in the results.
>>>>>>>>>>>>>>>>>>>>>> Bringing more companies on board will help, but I'm
>>>>>>>>>>>>>>>>>>>>>> still not sure that alone will make it independent 
>>>>>>>>>>>>>>>>>>>>>> Commercial companies can afford to dedicate staff to
>>>>>>>>>>>>>>>>>>>>>> improving Benchmark so that their products look better.
>>>>>>>>>>>>>>>>>>>>>> Open source projects just cant do that, so we are at a
>>>>>>>>>>>>>>>>>>>>>> distinct disadvantage.
>>>>>>>>>>>>>>>>>>>>>> Should we allow a commercially driven OWASP project
>>>>>>>>>>>>>>>>>>>>>> who's aim could be seen be to promote commercial 
>>>>>>>>>>>>>>>>>>>>>> If so, what sort of checks and balances does it need?
>>>>>>>>>>>>>>>>>>>>>> Those are the sort of questions I'd like an independent
>>>>>>>>>>>>>>>>>>>>>> review to look at.
>>>>>>>>>>>>>>>>>>>>>> I do think there are some immediate steps that could be
>>>>>>>>>>>>>>>>>>>>>> taken:
>>>>>>>>>>>>>>>>>>>>>> I'd like to see the Benchmark project page clearly
>>>>>>>>>>>>>>>>>>>>>> state thats its at a very early stage and that the 
results are _not_ yet
>>>>>>>>>>>>>>>>>>>>>> suitable for use in commercial literature.
>>>>>>>>>>>>>>>>>>>>>> I'd also like the main companies developing Benchmark
>>>>>>>>>>>>>>>>>>>>>> to be clearly stated on the main page. If and when other 
companies get
>>>>>>>>>>>>>>>>>>>>>> involved then this would actually help the project's 
claim of vendor
>>>>>>>>>>>>>>>>>>>>>> independence.
>>>>>>>>>>>>>>>>>>>>>> And I'd love to see a respected co-leader added to the
>>>>>>>>>>>>>>>>>>>>>> project who is not associated with any commercial or open 
source security
>>>>>>>>>>>>>>>>>>>>>> tools:)
>>>>>>>>>>>>>>>>>>>>>> And we should carry on discussing the project on this
>>>>>>>>>>>>>>>>>>>>>> list - I think such discussions are very healthy, and I'd 
love to see this
>>>>>>>>>>>>>>>>>>>>>> project mature to a state where it can be a trusted, 
independent and valued
>>>>>>>>>>>>>>>>>>>>>> resource.
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias
>>>>>>>>>>>>>>>>>>>>>> <tobias.gondrom at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>> @Simon:
>>>>>>>>>>>>>>>>>>>>>>> yes, the leaders list is the place for your
>>>>>>>>>>>>>>>>>>>>>>> discussions for project and chapter leaders
>>>>>>>>>>>>>>>>>>>>>>> @Timo: I like your framing of "Don't ask what OWASP
>>>>>>>>>>>>>>>>>>>>>>> can do for me, ask what I can do for OWASP."
>>>>>>>>>>>>>>>>>>>>>>> That should and is indeed the spirit of OWASP:-)
>>>>>>>>>>>>>>>>>>>>>>> Best regards, Tobias
>>>>>>>>>>>>>>>>>>>>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>>> ...
>>>>> [Message clipped]
>>>>> _______________________________________________
>>>>> Owasp-board mailing list
>>>>> Owasp-board at lists.owasp.org
>>>>> https://lists.owasp.org/mailman/listinfo/owasp-board
>>>> --
>>>> OWASP ZAP Project leader
>>>> _______________________________________________
>>>> OWASP-Leaders mailing list
>>>> OWASP-Leaders at lists.owasp.org
>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>> --
>> OWASP ZAP Project leader
>> _______________________________________________
>> OWASP-Leaders mailing list
>> OWASP-Leaders at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>OWASP-Leaders mailing list
>OWASP-Leaders at lists.owasp.org

More information about the OWASP-Leaders mailing list