[Owasp-leaders] [Owasp-board] OWASP Benchmark project - potential conflict of interest

jan.kopecky at owasp.org jan.kopecky at owasp.org
Tue Dec 1 15:56:26 UTC 2015

+1 Eoin, I think this is the best description of the problem.

Just my two cents - as many people believe OWASP benchmark project was abused, is there any official OWASP statement saying we really do not recommend Aspect/Contrast/Anyone ? I know that we have that known statement “OWASP does not promote any..” but here we are facing concrete issue which we should IMO address with concrete statement.

It would be really good to think about whether OWASP can even run projects as “Benchmark”. My personal opinion is that it cannot (or at least should not). Because we are trying hard to be vendor neutral as much as possible and when dealing with Benchmark results there will always be some winner. This winner will of course use it for marketing and we have the problem again.

Honestly if I had my company which offers security scanner and I would have the best score in "OWASP Benchmark” I would logically promote it as loudly as possible - if there is nothing stopping me from doing so. Do we have something like that in place?

My personal opinion is when OWASP makes any benchmark available then OWASP accidentally promotes the one with best score.

Thanks for reading!


From: Eoin Keary
Sent: ‎Tuesday‎, ‎December‎ ‎1‎, ‎2015 ‎10‎:‎16‎ ‎AM
To: Jeff Williams
Cc: owasp-leaders at lists.owasp.org

Hi Jeff,

Being with OWASP for 11 years, being a previous board vice chair and global board member for 5 years and having lead many OWASP projects such as the testing guide, code review guide etc I'd like to share how this is perceived....

My interpretation is the problem is that many folks see the OWASP benchmark run by Aspect/Contrast company staff and it happens the Contrast is top scoring. The second it was released as a project it was used in advertising and awareness campaigns. 

Second issue is that is its way to immature and does not have the sample data to be scientific and is simply used for commercial gain/product promotion.

It's not as if the project has more than 50% of the tool vendors /open source tools taking part such that the sample space stands up statistically. 

There is a perceived independence issue given the project was conceived and is lead by a tool vendor.

Many feel the project was conceived for the benefit of Contrast and it has lost credibility for that reason. 

The idea of the bench mark was/is good but it was rushed and taken advantage of without it being mature or proven.

It does not have the data to back up its claims and many OWASP people believe it was simply an advertising vehicle regardless of its future use, merit and adoption.

I believe this clarifies many peoples point of view.

Best regards,

Eoin Keary
OWASP Volunteer


On 1 Dec 2015, at 5:46 a.m., Jeff Williams <jeff.williams at owasp.org> wrote:

Hi Leaders,


I've refrained from commenting on the OWASP Benchmark project because I work for Contrast and used to work with Dave.  But before you judge, please hear my thoughts on this project, vendors, and OWASP in general.


As many of you know, I was the global chair of OWASP for over eight years. I set up the 501c3, created the chapters program, created the wiki, and led many successful projects. And I spent countless hours working with project leads, organizations, and vendors to teach them how OWASP works.


Contrast has heard your concerns and is already changing its marketing materials to emphasize that neither OWASP nor DHS endorses commercial products.  Both Contrast and I are huge OWASP supporters and we will do whatever we can to help achieve the mission. We want to make sure we are following the OWASP brand usage guidelines and will work with you to make sure we are. I wrote the original rules years ago and if it’s time to re-examine them then let’s change them in an open process.


The OWASP Benchmark is a real opportunity for OWASP to move the needle. I've spent considerable time with NSA’s Juliet and other similar test suites, and none are even close to the quality and ease of use of the OWASP Benchmark. I'd go so far as to say that if you haven't used the Benchmark, you really have no idea what the strengths and weaknesses of your tools are. The Benchmark isn't perfect. It could cover more vulns, but the ones in there are pretty damn important and the project is working on adding more vulnerability categories and more real world complexity. And the test cases aren't as complex as real code, but testing fundamental tool capabilities can tell us a lot.


The force-multiplier here is that by influencing tools and their vendors, OWASP can reach far more developers than it could with any number of awesome lists, cheat sheets, or standards. Do you wish tools were better at XXE, for example? If the OWASP community puts it in the Benchmark, it starts influencing buying behavior and the tools themselves.  The Benchmark puts OWASP in the driver's seat and it's already started to work.


On the leaders list, people have been making all kinds of assertions about marketing claims. But without a ruler, everyone is just guessing. This is exactly why we need the OWASP Benchmark. Without one, there’s no way to establish the truth of either the claims OR the counterclaims.  This ruler-free environment encourages vendors to make outrageous claims, like “we address the entire OWASP Top Ten” for example.  So, ironically, the Benchmark is the path towards more realistic marketing.


With any benchmark, some vendors aren't going to do well, and they will argue the ruler is broken.  Other vendors will do well and will promote their results.  As the ruler changes over time, vendor positions will change.  All of this is good for the community and for OWASP.  The best imaginable outcome would be to touch off an "arms-race" for accuracy in the appsec tools space.  The OWASP Benchmark will evolve and get better, the tools will get dragged along, and consumers will benefit through increased visibility.  This is exactly OWASP's mission, making appsec visible so that market forces can work.


On Mon, Nov 30, 2015 at 10:37 PM, Dave Wichers <dave.wichers at owasp.org> wrote:


Thanks so much for your post. For clarification, Justin refers to Coverity
numerous times throughout this post, but really means Contrast.


On 11/30/15, 4:55 PM, "Justin Searle" <justin at meeas.com> wrote:

>Psiinon, out of curiosity, from a purely project/code perspective,
>what do you feel would make the Benchmark tool more "independent"?
>From digging around in the source code, it seems they already have a
>fair number of report parsers for several OSS and commercial tools in
>their tree.  To keep this thread a bit more clean, perhaps post your
>response in the Benchmark project mailling list or in a GitHub issue
>then reply to this thread with link.  That will allow the Benchmark
>team to directly receive that feedback and allow those of use
>interested to follow up there to have the needed discussions to
>improve the tool.
>Eoin, Dinis, and Jim, it has been a while.  Sorry I've been so removed
>from this community the last couple of years.  From reading through
>the thread and articles, I don't think it is fair to say the board has
>been sitting on the issue.  From Josh's comments, it seems like the
>board did take action on this, however I understand if you disagreed
>with their decision.  You have always been some of the most vocal
>leaders here, and I applaud that.  Your contributions have been great
>to OWASP over the years, so please keep it up!
>Most of the concerns I've seen stated are around use of the OWASP
>brand in marketing.  I personally do not see merit in many of the
>concerns, however my personal greatest concerns are around statements
>like this:
>    "OWASP reports that the best static analysis tools score in the
>low 30¹s (out of 100) against this benchmark."
>That was in the Dark Reading article, and other statements were made
>like that in the Twitter video posted above.  That I feel is the most
>egregious of the brand misuse as it implies that OWASP as an
>organization has formally made statement.  Keep the full project name
>in tact such as "The OWASP Benchmark tool reports..." would be a much
>more accurate and less brand abusive way to make that statement, and
>based on current OWASP policies, much more inline with what is
>permitted.  While OWASP members feel even that is going too far,
>please remember that OWASP does not own the vast majority of the OWASP
>projects.  Each OWASP project is usually owned by the author and in
>many cases, any contributor to that project.  OWASP as an legal entity
>owns a very small percentage (none that I can even name off the top of
>my head).
>A few years ago when we were initially working on the new project
>leaders handbook and project roadmap (before OWASP disbanded all the
>global committees in 2013, including the Global Projects Committee),
>we discussed chaining the official verbiage to and OWASP "sponsored"
>project.  I think it is unfortunate we didn't codify that terminology,
>but regardless, I think that is the always a good way to think of
>OWASP projects: projects owned and run by individuals of the
>community.  However, not everyone in the OWASP community understand
>this.  This is easy to see in such statements as "Allowing this
>project to exist without ..." and forcing a "project be opened up to
>commits via Git so that outsiders can push commits to it" and OWASP
>should "decide on the future of this project".  Personally, I think
>the most of the drama around OWASP projects comes from this
>misunderstanding and OWASP community members trying to manage an OWASP
>project that OWASP doesn't own.
>However one of the most difficult issues that perpetuates this problem
>and in many cases conflicts concerning brand abuse is project naming.
>Since OWASP currently allows projects to use the OWASP name in their
>project name (which I think is a mistake), it is hard to refer to a
>project without in some way evoking the OWASP brand.  There is very
>little legal recourse in most countries to state a fact that "tool X
>generated score Y for product Z" in their marketing literature.  That
>does not imply that tool X promotes product Z.  And if tool X happens
>to be named OWASP Benchmark, then that is not brand infringement in
>most countries.  If this is a concern to the OWASP community, then the
>better recourse would be to reconsider OWASP's permission to allow
>projects to use OWASP in their project name.
>Preventing people from making such statements is usually handled in a
>EULA saying how you can and can't use the tool and the tool output,
>which in most cases including this would be contrary from the official
>OSS definition and most OSS licenses.  So the best distance I think we
>could hope to obtain is to disallow the use of OWASP in any project
>And by the way, why would we ever want to stop ANY company out there
>from using OWASP tools and documentation?  Why would we ever want ANY
>company NOT to advertise that they use OWASP tools and documentation?
>Why would we NOT want a company to state they they use OWASP tools in
>their marketing literature?  As long as it is clear that OWASP does
>not endorse that company, we should encourage the spread and use of
>OWASP tools.  Does anyone have a problem with saying website Z has
>been tested for all OWASP Top 10 risks in their marketing literature?
>What about saying that all vulnerabilities identified by OWASP Zed
>Attack Proxy Project have been remediated in website Z literature?
>What about all the current DAST/SAST tools that have an "OWASP Top 10"
>testing mode?  I don't think any of these imply that a project is
>endorsed by OWASP, but if this is a concern for people, would ..."all
>vulnerabilities identified by Zed Attack Proxy Project have been
>remediated" be better?
>As for a Jeff (or his company) using the benchmark scores from his own
>OWASP sponsored project in marketing literature to help customers
>understand their commercial offering, I have no qualms with that.  I
>don't find that a breach of trust or brand abuse.  I only see brand
>abuse in statements mentioned above that stated "Owasp found..." and
>such where the tool name was not used, which is explicitly stating a
>false OWASP perspective.  Jeff and Coverity in benchmarking their tool
>against their own opensource project simply ties the two together in
>such a way that can be tested.  Based on statements made by Psiinon
>and others, including Coverity's competition, the tool works and does
>not seem to be skewed towards Coverity's tool, even though they score
>the highest.  The tool is opensource.  If anyone believes the tool
>unfairly scores Coverity's tool, or doesn't not provide benefit to
>other assessment tools who want to improve their scanning engines,
>please dig through the code and identify how it does that.  All I've
>seen so far is people disagreeing with how the metric is generated and
>the number of tests involved, which in itself doesn't seem to portray
>bias for one tool over another.  If the tool is found to favor
>Coverity's scanning tool, then that will be shown by someone with time
>and interest, and if that is the case, the brand loss will by
>Coverity's, not OWASP's.
>As for actions, I agree with the actions the board seems to have made
>so far.  I do not think any penalties be levied against the Benchmark
>project.  I do not think that they should be downgrade back to
>incubator, which seems a petty and meaningless action to me.  The
>maturity seems to say Lab quality more than many other existing Lab
>projects.  I do not think OWASP has any right or reason to force the
>Benchmark project to allow commits from additional persons.  Having a
>single person do actual commits to main trunks while other offer pull
>requests is common and very standard in OSS, and in now way portrays
>how "open" the project community is.  And banning companies from any
>mention OWASP projects in marketing efforts, wether project leaders
>are associate with said companies or not, would be foolish in our
>efforts to growing OWASP brand, as long as such marketing efforts do
>not implicitly or explicitly imply OWASP endorsement of a company, its
>tools, or its services.
>As for my suggestions to the OWASP board, I'd recommend the following:
> - An official statement on the Benchmark project page at owasp.org
>stating as Johanna suggested, that OWASP does not endorse any company,
>commercial tool, or commercial service
> - A request to Coverty to make a similar statement on their website
>and future marketing efforts just to clarify this misunderstanding
> - A formal cease and desist letter to Coverty to stop making explicit
>claims in OWASP's behalf such as "OWASP found ..." and to restrict all
>use of the term "OWASP" as part of the "OWASP Benchmark" project's
>formal name.
>As for the Benchmark project, I'd recommend the following:
> - If the tool doesn't already do so, I'd recommend a simply statement
>in the Benchmark reports saying that scores from the tool does not
>imply any endorsement for or against any tool tested.
> - Also, digging through your sourcecode tree, I noticed there doesn't
>seem to be any copyright notices.  I'd recommend you adding those
>copyright notices to whoever owns the code, otherwise it is hard to
>enforce any copyright license restrictions.
>And finally, as for the OWASP community, I'd encourage you to decide
>if it makes sense to remove the ability of projects to formally use
>OWASP as part of their project name.  If we don't do this then
>individual project brand and OWASP brand becomes commingled, and
>ownership becomes less clear.  Project leaders, for your current or
>future projects, I'd personally recommend you don't use OWASP in the
>title so you can build project brand recognition independent of OWASP,
>and instead do something like "Project X, an OWASP project" in your
>project marketing.
>Justin Searle
>Managing Partner - UtiliSec
>+1 801-784-2052
>justin at utilisec.com
>justin at meeas.com
>On Mon, Nov 30, 2015 at 10:40 AM, psiinon <psiinon at gmail.com> wrote:
>>> In the short term, It is better to remove it from OWASP, leaving the
>>> open for its return (in a future when some of the independence and
>>> issues have been solved)
>> I would be delighted to welcome an independent, high quality Benchmark
>> project back into OWASP :)
>>> Specially when recently we made David Rook remove this much more benign
>>> 'commercial content' from OWASP
>>> Dinis
>>> On 30 November 2015 at 17:17, psiinon <psiinon at gmail.com> wrote:
>>>> I'd like to start by saying that I actually _like_ the Benchmark
>>>> Myself and other ZAP developers have made some contributions to it,
>>>> we have used (and will continue to use) it to make ZAP better.
>>>> I think these sort of testing applications are very valuable to all
>>>> security tools, and I'd like to thank Dave and his team for the
>>>> amount of effort involved in developing and open sourcing it.
>>>> But I dont think it should be an OWASP project.
>>>> I do not think that a vendor led project can ever objectively evaluate
>>>> competing commercial and open source projects.
>>>> I do not think that just saying 'pull requests welcomed' makes a
>>>> vendor neutral.
>>>> I do not think that a project as mired in controversy as the Benchmark
>>>> project can ever recover to become truly independent.
>>>> I am very disappointed in the Boards handling of this affair.
>>>> Ideally I'd like Dave to understand how much damage this project has
>>>> and to withdraw it as an OWASP project, while still maintaining it as
>>>>a very
>>>> valuable vendor led open source resource.
>>>> Failing that I really hope that the Board comes to its senses and
>>>> the Benchmark project before even more damage is done.
>>>> At the _very_ least it should flag the project as being 'in dispute'
>>>> Kevin suggested) while a more detailed evaluation is performed.
>>>> However I'm rapidly loosing loosing faith that the Board will do the
>>>> right thing and protect OWASP's image in the way that they should have
>>>> already done.
>>>> Members - please make your voices heard before more people and
>>>> leave OWASP.
>>>> Simon
>>>> On Sat, Nov 28, 2015 at 5:14 AM, Jim Manico <jim.manico at owasp.org>
>>>>> WAFEC does not "do vendor assessment"; they define a comprehensive
>>>>> standard built by many vendors and let the community use that
>>>>>standard to
>>>>> measure tools on their own. Just a FYI, I was involved in the early
>>>>> of this project. (Things may have changed since my involvement, I'm
>>>>> Tony has more details here)
>>>>> Johanna's comments on this issue lead me to believe that the damage
>>>>> to both OWASP and DHS is even more destructive that I thought. It
>>>>>saddens me
>>>>> to see this level of abuse just to sell product.
>>>>> --
>>>>> Jim Manico
>>>>> Global Board Member
>>>>> OWASP Foundation
>>>>> https://www.owasp.org
>>>>> Join me in Rome for AppSecEU 2016!
>>>>> On Nov 28, 2015, at 2:40 AM, Josh Sokol <josh.sokol at owasp.org> wrote:
>>>>> One of the ideas that Andrew proposed was actually approaching WAFEC
>>>>> learn more about how they do vendor assessment in a neutral way.
>>>>>It's great
>>>>> to hear that we have a resource here already that we can leverage.
>>>>>I wasn't
>>>>> aware of your affiliation.
>>>>> ~josh
>>>>> On Nov 27, 2015 2:47 PM, "Tony Turner" <tony.turner at owasp.org> wrote:
>>>>>> I sincerely hope so. That's not the impression I got from others
>>>>>> comments. Personally I haven't used the tool at all, but as I'm the
>>>>>> lead for another product evaluation project (WAFEC) I'm very
>>>>>>sensitive to
>>>>>> the need of collaboration with many different vendors. There really
>>>>>>has to
>>>>>> be a very high level (almost paranoid level) transparency with how
>>>>>> are approached, worked with, how requirements for evaluation are
>>>>>> and how metrics are derived.
>>>>>> It appears the project team is attempting to address these last 2
>>>>>> somewhat but I'd like to see more specifics, and the lack of
>>>>>>information on
>>>>>> how they are addressing vendor communication, participation and
>>>>>> seems a bit concerning. Lastly, it is my opinion that project
>>>>>> should not belong to anyone working for or with a
>>>>>> stake for any vendor being evaluated. I think this is a flawed
>>>>>>model and
>>>>>> should transition to a vendor neutral party.
>>>>>> On Nov 27, 2015 3:16 PM, "Josh Sokol" <josh.sokol at owasp.org> wrote:
>>>>>>> I don't know what qualifies as "significant" in your mind, but my
>>>>>>> understanding is that there have been contributions from other
>>>>>>> https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements
>>>>>>> Still, Dave would like more, but he can't force them to help.
>>>>>>> ~josh
>>>>>>> On Fri, Nov 27, 2015 at 1:45 PM, Tony Turner
>>>>>>><tony.turner at owasp.org>
>>>>>>> wrote:
>>>>>>>> While I can appreciate that they started with Contrast, if there
>>>>>>>> hasn't been significant effort to include other vendors it's a
>>>>>>>> benchmark. It's easy to state you haven't gotten support from
>>>>>>>>other vendors
>>>>>>>> and that's fine, but until you do there's really nothing to
>>>>>>>>release. Why was
>>>>>>>> it ever upgraded? Talking about the results without an accurate
>>>>>>>> analysis is akin to snake oil.
>>>>>>>> On Nov 27, 2015 1:49 PM, "Josh Sokol" <josh.sokol at owasp.org>
>>>>>>>>> Thank you for the links to those articles.  The first one
>>>>>>>>> the strengths and weaknesses of the different methods of
>>>>>>>>>evaluating for
>>>>>>>>> application vulnerabilities.  The section on the Benchmark seems
>>>>>>>>> appropriate to me.  That seems like an excellent description of
>>>>>>>>>what the
>>>>>>>>> project is designed to do.  I see some metrics in there about
>>>>>>>>>which tools
>>>>>>>>> are more effective on which types of vulnerabilities, but I
>>>>>>>>>don't see him
>>>>>>>>> straight up saying "The OWASP Benchmark proves that Contrast is
>>>>>>>>> This seems like statements made based on some level of testing
>>>>>>>>>and research.
>>>>>>>>> Honestly, I don't see any OWASP brand abuse in that article.
>>>>>>>>>Whether it's
>>>>>>>>> in good taste or not at this stage in the project is certainly
>>>>>>>>> but if you look at the brand usage guidelines
>>>>>>>>> I don't see any violations.  We need to govern to policy here
>>>>>>>>>which is why
>>>>>>>>> Paul and Noreen are evaluating changes to the guidelines and our
>>>>>>>>> policies to make abuse more difficult.
>>>>>>>>> The second article is a competing vendor's reaction to the first.
>>>>>>>>> He makes some good points about the issues with Benchmark, but
>>>>>>>>>he also says
>>>>>>>>> that he hopes that it will be improved over time, and Dave has
>>>>>>>>>committed to
>>>>>>>>> that.  What I don't see is the vendor saying "...and Veracode
>>>>>>>>>has committed
>>>>>>>>> resources to help make the Benchmark more accurate across all
>>>>>>>>>tool sets".
>>>>>>>>> The Benchmark page is pretty clear that it does it's best to
>>>>>>>>>provide a
>>>>>>>>> benchmark without working exactly like a real-world application.
>>>>>>>>> Maybe some
>>>>>>>>> more disclaimer text about where the project is at today would
>>>>>>>>>be in order
>>>>>>>>> to validate some of Chris' concerns, but I hardly see this as
>>>>>>>>>"brand abuse"
>>>>>>>>> or a reason to demote the project.
>>>>>>>>> Please consider that I have spoken with both Dave and Jeff on
>>>>>>>>> topic and read much of the discussions around it before
>>>>>>>>>formulating my
>>>>>>>>> opinion.  I doubt that you have done the same so I'm not sure
>>>>>>>>>how you can
>>>>>>>>> claim that you have researched the issues and all parties
>>>>>>>>>involved when you
>>>>>>>>> haven't even spoken with the two people whom you are accusing of
>>>>>>>>> impropriety.  I have no bias here.  I am simply speaking with the
>>>>>>>>> individuals involved, looking at the currently OWASP policies and
>>>>>>>>> guidelines, and helping to determine our next steps.
>>>>>>>>> ~josh
>>>>>>>>> On Fri, Nov 27, 2015 at 12:22 PM, johanna curiel curiel
>>>>>>>>> <johanna.curiel at owasp.org> wrote:
>>>>>>>>>> >>While I agree with you that there has been some brand abuse,
>>>>>>>>>> >> was abuse by Contrast (specifically their marketing
>>>>>>>>>>department), and not by
>>>>>>>>>> >> "these gentlemen" as  you state.
>>>>>>>>>> Really? ..'some brand abuse'..this is more than brand abuse
>>>>>>>>>> Josh , please read also the article written by Jeff
>>>>>>>>>> And Veracode's reaction including others in Twitter
>>>>>>>>>> My strong advice is to research the issues and all the parties
>>>>>>>>>> involved before making statements
>>>>>>>>>> On Fri, Nov 27, 2015 at 2:07 PM, Josh Sokol
>>>>>>>>>><josh.sokol at owasp.org>
>>>>>>>>>> wrote:
>>>>>>>>>>> Jim,
>>>>>>>>>>> A concern was expressed to the Board and, frankly, I am
>>>>>>>>>>> by you saying that this was "brushed under the rug".  The
>>>>>>>>>>>Board delegated
>>>>>>>>>>> Matt to talk with Dave and they had a lengthy conversation on
>>>>>>>>>>>the subject.
>>>>>>>>>>> The Board delegated me to talk with Jeff and we had a lengthy
>>>>>>>>>>> on the subject.  If you do not trust in our abilities to read
>>>>>>>>>>>people, ask
>>>>>>>>>>> the right questions, and provide honest feedback about our
>>>>>>>>>>> then that's a bigger issue that we should take offline.  After
>>>>>>>>>>> conversations, we took the time to call a special two-hour
>>>>>>>>>>>session of the
>>>>>>>>>>> Board in order to discuss this subject (and only this
>>>>>>>>>>>subject).  We spoke
>>>>>>>>>>> about all facets of the issue at hand, about the challenges
>>>>>>>>>>>and possible
>>>>>>>>>>> solutions, and concluded on some very concrete next steps.
>>>>>>>>>>> While I agree with you that there has been some brand abuse, it
>>>>>>>>>>> was abuse by Contrast (specifically their marketing
>>>>>>>>>>>department), and not by
>>>>>>>>>>> "these gentlemen" as  you state.  Unless you can point to some
>>>>>>>>>>>sort of
>>>>>>>>>>> evidence showing that Jeff and/or Dave first-hand abused the
>>>>>>>>>>>brand, then I
>>>>>>>>>>> believe that you are speaking with your heart instead of with
>>>>>>>>>>>your head.  I
>>>>>>>>>>> appreciate your passion, but I label this as conspiracy theory
>>>>>>>>>>> without evidence to support your claims, I cannot accept it as
>>>>>>>>>>> other.
>>>>>>>>>>> ~josh
>>>>>>>>>>> On Fri, Nov 27, 2015 at 11:39 AM, Jim Manico
>>>>>>>>>>> <jim.manico at owasp.org> wrote:
>>>>>>>>>>>> Josh,
>>>>>>>>>>>> I stand by my comments and perspective, but I'm disheartened
>>>>>>>>>>>> you consider my presentation of facts (and the concerns of
>>>>>>>>>>>>many active
>>>>>>>>>>>> members of our community) as a "conspiracy theory".
>>>>>>>>>>>> In my experience, these kind of comments border on insults and
>>>>>>>>>>>> only cause folks to harden their opinions.
>>>>>>>>>>>> Once again I feel these gentlemen got away with a kind of
>>>>>>>>>>>> abuse that is very hurtful to the OWASP community but I am at
>>>>>>>>>>>>a loss as to
>>>>>>>>>>>> how handle or prevent these kinds of mishaps - especially
>>>>>>>>>>>>when board members
>>>>>>>>>>>> like yourself seem willing to - from what I see - brush it
>>>>>>>>>>>>under the rug.
>>>>>>>>>>>> --
>>>>>>>>>>>> Jim Manico
>>>>>>>>>>>> Global Board Member
>>>>>>>>>>>> OWASP Foundation
>>>>>>>>>>>> https://www.owasp.org
>>>>>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>>>>> On Nov 27, 2015, at 7:23 PM, Josh Sokol <josh.sokol at owasp.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Admittedly, this was my gut reaction at first as well.  I
>>>>>>>>>>>> linking all of these companies, people, and projects together
>>>>>>>>>>>>in my mind
>>>>>>>>>>>> (there are some loose links there) and painted a big
>>>>>>>>>>>>conspiracy picture
>>>>>>>>>>>> similar to what Jim and Dinis have stated.  But, after
>>>>>>>>>>>>speaking directly
>>>>>>>>>>>> with Jeff, and hearing about the conversation that Dave and
>>>>>>>>>>>>Matt had, I've
>>>>>>>>>>>> changed my mind.
>>>>>>>>>>>> I think it begins with the project itself.  If you aren't
>>>>>>>>>>>>sold on
>>>>>>>>>>>> the idea of the Benchmark, then you'll never be able to get
>>>>>>>>>>>>to the same
>>>>>>>>>>>> place.  My original line of thinking was that it was just a
>>>>>>>>>>>>bar for vendors
>>>>>>>>>>>> to compare their tools against eachother, but that's a bit
>>>>>>>>>>>>myopic.  We are
>>>>>>>>>>>> in an industry where things evolve very quickly.  As a
>>>>>>>>>>>>customer of these
>>>>>>>>>>>> tools, I know firsthand that something that a tool does today
>>>>>>>>>>>>may not be the
>>>>>>>>>>>> case a week from now.  Likewise, new features are being added
>>>>>>>>>>>>daily and I
>>>>>>>>>>>> need a point-in-time metric to be able to gauge continual
>>>>>>>>>>>> Cool, right?  But not a game changer.  The game changer part
>>>>>>>>>>>>comes when you
>>>>>>>>>>>> realize that by developing and evolving the tests that go
>>>>>>>>>>>>into the
>>>>>>>>>>>> Benchmark, we are moving the bar higher and higher.  We
>>>>>>>>>>>>(OWASP) are
>>>>>>>>>>>> effectively setting the standard by which these tools will be
>>>>>>>>>>>>compared.  A
>>>>>>>>>>>> tool that receives a lower score on the Benchmark today knows
>>>>>>>>>>>>exactly what
>>>>>>>>>>>> they need to work on in order to pass that test tomorrow and
>>>>>>>>>>>>we already have
>>>>>>>>>>>> examples of tools that have made improvements because of
>>>>>>>>>>>>their Benchmark
>>>>>>>>>>>> score (Ask Simon about ZAP's experience with the Benchmark).
>>>>>>>>>>>>I don't think
>>>>>>>>>>>> that anyone can argue that the Benchmark project isn't being
>>>>>>>>>>>>effective when
>>>>>>>>>>>> OWASP's own tools are being driven forward as a result of
>>>>>>>>>>>>using it.
>>>>>>>>>>>> But, but, but, Dave and Jeff own Aspect and have stock in
>>>>>>>>>>>> Contrast and Jeff is the Contrast CTO and Contrast got good
>>>>>>>>>>>>scores so it's a
>>>>>>>>>>>> conspiracy right?  Is there some code that allows Contrast to
>>>>>>>>>>>>use the
>>>>>>>>>>>> Benchmark?  Absolutely.  Can you really blame Dave for
>>>>>>>>>>>>starting his testing
>>>>>>>>>>>> on the effectiveness of the Benchmark with a tool that he
>>>>>>>>>>>>owned and is
>>>>>>>>>>>> familiar with?  If I were going to start a similar project,
>>>>>>>>>>>>there's no
>>>>>>>>>>>> question in my mind that I would begin my testing with the
>>>>>>>>>>>>tools that I have
>>>>>>>>>>>> available to me.  That said, is there code that allows other
>>>>>>>>>>>>tools to use
>>>>>>>>>>>> the Benchmark?  Absolutely.
>>>>>>>>>>>> Regarding "Dave has a history of breaching his duty to be
>>>>>>>>>>>> neutral", while I cannot comment on his past actions, I can
>>>>>>>>>>>>judge what we've
>>>>>>>>>>>> seen recently.  Matt saw a presentation from Dave on the
>>>>>>>>>>>>Benchmark at a
>>>>>>>>>>>> conference in Chicago.  He said that he felt that the message
>>>>>>>>>>>> appropriate and while IAST tools were mentioned as receiving
>>>>>>>>>>>>higher scores,
>>>>>>>>>>>> it wasn't a "Contrast is the best" type of message, more of a
>>>>>>>>>>>>generality.  I
>>>>>>>>>>>> saw a very similar (if not the same) talk by Jeff at LASCON
>>>>>>>>>>>>2015 and the
>>>>>>>>>>>> message was exactly the same.  I watched the talk expecting
>>>>>>>>>>>>some sort of
>>>>>>>>>>>> impropriety, but found none.  So, perhaps Dave has abused
>>>>>>>>>>>>some privilege
>>>>>>>>>>>> granted to him in the past, but what I've seen from him at
>>>>>>>>>>>>this point, with
>>>>>>>>>>>> respect to the Benchmark, has been appropriate.
>>>>>>>>>>>> You have a very good point with respect to the Contrast
>>>>>>>>>>>> message around the Benchmark.  It's been completely absurd,
>>>>>>>>>>>>over the top,
>>>>>>>>>>>> and, in my personal opinion, intolerable.  In fact, I
>>>>>>>>>>>>experienced the same
>>>>>>>>>>>> thing that you talked about with them at LASCON 2015 where
>>>>>>>>>>>>they stood in
>>>>>>>>>>>> front of the door of the room Jeff was speaking in and
>>>>>>>>>>>>scanned attendees as
>>>>>>>>>>>> they went into the talk.  I agree that these types of
>>>>>>>>>>>>aggressive marketing
>>>>>>>>>>>> tactics cannot be tolerated at OWASP.  In addition, we have
>>>>>>>>>>>>seen several
>>>>>>>>>>>> marketing messages from them effectively implying that OWASP
>>>>>>>>>>>> Contrast.  Clearly this is not OK.  I've spoken with Jeff
>>>>>>>>>>>>about it and we
>>>>>>>>>>>> agreed that it is not in the Benchmark's best interest to
>>>>>>>>>>>>have this
>>>>>>>>>>>> aggressive Contrast marketing around it at such an early
>>>>>>>>>>>>stage.  He has said
>>>>>>>>>>>> that he is not responsible for Contrast's marketing team, but
>>>>>>>>>>>>that he would
>>>>>>>>>>>> speak with the people who are.  I haven't seen a single
>>>>>>>>>>>>message from them
>>>>>>>>>>>> since so I'm guessing that he's made good on this promise.
>>>>>>>>>>>>While that's an
>>>>>>>>>>>> excellent start, OWASP's takeaway here should be that we need
>>>>>>>>>>>>to do a better
>>>>>>>>>>>> job with our brand usage guidelines both in terms of the
>>>>>>>>>>>>wording and
>>>>>>>>>>>> enforcement.  There are many other companies out there that
>>>>>>>>>>>>use the OWASP
>>>>>>>>>>>> brand and I think that we agree that selective enforcement
>>>>>>>>>>>>against Contrast
>>>>>>>>>>>> is not the right answer.  Paul and Noreen are actively
>>>>>>>>>>>>working on this.
>>>>>>>>>>>> Either way, I think that implying that activities from a
>>>>>>>>>>>>vendor's marketing
>>>>>>>>>>>> department means that the project is not objective is not
>>>>>>>>>>>>inappropriate.  If
>>>>>>>>>>>> we feel that the project is not objective, then separate
>>>>>>>>>>>>measures need to be
>>>>>>>>>>>> taken to drive contribution diversity into it.  That I
>>>>>>>>>>>>absolutely agree with
>>>>>>>>>>>> and the message from Dave was that he would love to have more
>>>>>>>>>>>> to his project.  But, seeing as we cannot force people to
>>>>>>>>>>>>work on it, this
>>>>>>>>>>>> becomes a matter of "put up or shut up".  The same goes for
>>>>>>>>>>>>the experts that
>>>>>>>>>>>> you said reviewed the code.  If they feel that it is somehow
>>>>>>>>>>>>skewed towards
>>>>>>>>>>>> Contrast, they have the power to change that.  Now, if
>>>>>>>>>>>>someone tries to
>>>>>>>>>>>> participate and Dave tells them "No thanks", then I agree we
>>>>>>>>>>>>have a problem,
>>>>>>>>>>>> but I don't hear anyone inferring that happened.
>>>>>>>>>>>> Please, let's drop the conspiracy theories and focus on the
>>>>>>>>>>>> tangible things that we can do to help an OWASP project to be
>>>>>>>>>>>> successful.  Help find more participants to drive diversity,
>>>>>>>>>>>>update our
>>>>>>>>>>>> brand usage guidelines to prevent abuse, enforce them widely,
>>>>>>>>>>>>etc.  Thank
>>>>>>>>>>>> you.
>>>>>>>>>>>> ~josh
>>>>>>>>>>>> On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico
>>>>>>>>>>>> <jim.manico at owasp.org> wrote:
>>>>>>>>>>>>> Dinis,
>>>>>>>>>>>>> Like a rare celestial moment when all the planets plus Pluto
>>>>>>>>>>>>> aligned, I just read your email on the future of OWASP
>>>>>>>>>>>>>projects thinking,
>>>>>>>>>>>>> "Dinis is spot on".
>>>>>>>>>>>>> Reflecting on projects I manage or work on...
>>>>>>>>>>>>> The Java Encoder and HTML Sanitizer are likely best moved to
>>>>>>>>>>>>> Apache now that they have reached a measure of adoption and
>>>>>>>>>>>>>maturity. Apache
>>>>>>>>>>>>> would be a much better long term custodian. Perhaps the same
>>>>>>>>>>>>>for AppSensor,
>>>>>>>>>>>>> but not my project - just thinking out loud.
>>>>>>>>>>>>> Other similar defensive projects are still being noodled on,
>>>>>>>>>>>>> OWASP is a decent home for these research efforts.
>>>>>>>>>>>>> The whole tools category is also something to consider.
>>>>>>>>>>>>> Dependency Check and of course ZAP are some of the best
>>>>>>>>>>>>>projects that OWASP
>>>>>>>>>>>>> offers, are they best served where they are today? Both have
>>>>>>>>>>>>> communities of developers but I don't see the foundation
>>>>>>>>>>>>>doing much to
>>>>>>>>>>>>> support these efforts.
>>>>>>>>>>>>> ASVS has the opportunity to effect massive change, I would to
>>>>>>>>>>>>> love to see major investment and volunteer activity here.
>>>>>>>>>>>>>Pro tech writer,
>>>>>>>>>>>>> detailed discourses on each individual requirement, etc. If
>>>>>>>>>>>>>I was king (and
>>>>>>>>>>>>> I am not, at all) I would invest in ASVS on a 6 figure
>>>>>>>>>>>>>scale. (And who
>>>>>>>>>>>>> started ASVS? Jeff, Dave and Boberski, hat tip to such a
>>>>>>>>>>>>>marvelous idea). Or
>>>>>>>>>>>>> maybe moving ASVS to the W3C or IETF would help it grow?
>>>>>>>>>>>>> The Proactive Controls was a pet project but as we approach
>>>>>>>>>>>>> we have several active/awesome volunteers working on it. We
>>>>>>>>>>>>>will be making
>>>>>>>>>>>>> the doc "world editable" to make contributions easy. OWASP
>>>>>>>>>>>>>seems like a good
>>>>>>>>>>>>> home for such an awareness doc. Same with T10, especially if
>>>>>>>>>>>>>community edits
>>>>>>>>>>>>> are welcome.
>>>>>>>>>>>>> Anyhow, I'm with you on this Dinis. Once a project starts to
>>>>>>>>>>>>> reach production quality, spinning off the project as an
>>>>>>>>>>>>>external project or
>>>>>>>>>>>>> moving it to a different foundation where managing
>>>>>>>>>>>>>production software or
>>>>>>>>>>>>> formal standards is their thing seems realistic.
>>>>>>>>>>>>> I don't have all the answers here, but your email certainly
>>>>>>>>>>>>> resonated with me.
>>>>>>>>>>>>> Aloha,
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Jim Manico
>>>>>>>>>>>>> Global Board Member
>>>>>>>>>>>>> OWASP Foundation
>>>>>>>>>>>>> https://www.owasp.org
>>>>>>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>>>>>> On Nov 26, 2015, at 11:26 PM, Dinis Cruz
>>>>>>>>>>>>><dinis.cruz at owasp.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Jim's reading of this situation is exactly my view on the
>>>>>>>>>>>>> of the Contrast tool and how it has been 'pushing' the rules
>>>>>>>>>>>>>of engagement
>>>>>>>>>>>>> to an very 'fuzzy' moral/ethical/commercial limit :)
>>>>>>>>>>>>> As per my last email, a key problem here is the 'perceived
>>>>>>>>>>>>> expectation' of what is an OWASP project, and how it should
>>>>>>>>>>>>>be consumed.
>>>>>>>>>>>>> If you look at the OWASP benchmark as a research project,
>>>>>>>>>>>>> the only way it could be making the kind of claims it makes
>>>>>>>>>>>>>(and have
>>>>>>>>>>>>> credibility) is if it had evolved from OWASP, with its own
>>>>>>>>>>>>> community
>>>>>>>>>>>>> On 26 November 2015 at 21:01, Jim Manico
>>>>>>>>>>>>><jim.manico at owasp.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I have a different take on this situation but my opinion is
>>>>>>>>>>>>>> "minority opinion". I will respect the rest of the boards
>>>>>>>>>>>>>>take on this, but
>>>>>>>>>>>>>> here is how I see it.
>>>>>>>>>>>>>> First of all, Jeff has stated that he feels I am attacking
>>>>>>>>>>>>>> personally from a past personal grudge, and frankly I do
>>>>>>>>>>>>>>not fault him for
>>>>>>>>>>>>>> that perspective since we definitely have history with
>>>>>>>>>>>>>>conflict. So it's
>>>>>>>>>>>>>> fair to take my opinion on this with a grain of salt.
>>>>>>>>>>>>>> I look at this situation from the perspective of a forensic
>>>>>>>>>>>>>> investigator.
>>>>>>>>>>>>>> 1) The Benchmark project had Contrast hooks and only
>>>>>>>>>>>>>> hooks in it when I reviewed it so this leads me to believe
>>>>>>>>>>>>>>that the project
>>>>>>>>>>>>>> was clearly built with Contrast in mind from the ground up,
>>>>>>>>>>>>>>at least in some
>>>>>>>>>>>>>> way.
>>>>>>>>>>>>>> 3) Dave has a history of breaching his duty to be vendor
>>>>>>>>>>>>>> neutral. He was gifted with a keynote in South Korea a few
>>>>>>>>>>>>>>years ago, and
>>>>>>>>>>>>>> used that opportunity to discuss and pitch Contrast, on
>>>>>>>>>>>>>>stage, during a
>>>>>>>>>>>>>> keynote - with Contrast specific slides. This is just
>>>>>>>>>>>>>>supporting evidence of
>>>>>>>>>>>>>> his intention at OWASP to push Contrast in ways that I
>>>>>>>>>>>>>>think are against the
>>>>>>>>>>>>>> intentions and goals of our foundation.
>>>>>>>>>>>>>> 3) Other experts have reviewed the project and felt that
>>>>>>>>>>>>>> of the tests were very slanted and almost contrived to
>>>>>>>>>>>>>>support Contrast. I
>>>>>>>>>>>>>> can drag those folks into this conversation, but I do not
>>>>>>>>>>>>>>think that would
>>>>>>>>>>>>>> help in any way. So it's fair to call this point heresy.
>>>>>>>>>>>>>> 4) I do not see this project as revolutionary, at all. Every
>>>>>>>>>>>>>> vendor has their own test suite tuned for their tool. As
>>>>>>>>>>>>>>the benchmark
>>>>>>>>>>>>>> stands today, I see it as just another vendors
>>>>>>>>>>>>>>product-specific benchmark.
>>>>>>>>>>>>>> Mass collaboration from many vendors is not just a "nice to
>>>>>>>>>>>>>>have" but a base
>>>>>>>>>>>>>> requirement to get even close to useful for objective tool
>>>>>>>>>>>>>> 5) Jeff stating that his Marketing people went over the
>>>>>>>>>>>>>>line is
>>>>>>>>>>>>>> also an admission that - well, they went over the line. By
>>>>>>>>>>>>>>the same token
>>>>>>>>>>>>>> Jeff was in his booth at AppSec USA surrounded by benchmark
>>>>>>>>>>>>>> material, discussing this to prospects and he even asked me
>>>>>>>>>>>>>>and Mr Coates to
>>>>>>>>>>>>>> wade into this debate and support Dave. So to say he was
>>>>>>>>>>>>>>not involved and it
>>>>>>>>>>>>>> was only his marketing people seems a stretch at best.
>>>>>>>>>>>>>> 6) The Contrast marketing team was wandering around the
>>>>>>>>>>>>>> conference zapping folks to get leads, and I asked them to
>>>>>>>>>>>>>>stay in their
>>>>>>>>>>>>>> booth, which is standard conference policy. These folks
>>>>>>>>>>>>>>know better but are
>>>>>>>>>>>>>> again going over the line to sell product at OWASP. There
>>>>>>>>>>>>>>is a better way
>>>>>>>>>>>>>> (like focusing on product capability and language support,
>>>>>>>>>>>>>>have consistent +
>>>>>>>>>>>>>> stellar customer service, have a humble and gracious
>>>>>>>>>>>>>>attitude to all
>>>>>>>>>>>>>> prospects and customers, actively participate in OWASP in a
>>>>>>>>>>>>>>vendor neutral
>>>>>>>>>>>>>> and community supportive way, etc).
>>>>>>>>>>>>>> Please note, I think Contrast is a decent tool, I've
>>>>>>>>>>>>>>offered to
>>>>>>>>>>>>>> resell in the past, and I have recommended it in certain
>>>>>>>>>>>>>>situations - even
>>>>>>>>>>>>>> after this situation arose. I'm stating this out of
>>>>>>>>>>>>>>honestly and desire to
>>>>>>>>>>>>>> put my cards on the table. I truly want Jeff and Dave to be
>>>>>>>>>>>>>>successful. They
>>>>>>>>>>>>>> have dedicated their lives to AppSec and if anyone should
>>>>>>>>>>>>>>win big-time, I
>>>>>>>>>>>>>> hope it's them. I even told Jeff I hope he hits the mother
>>>>>>>>>>>>>>load and donates
>>>>>>>>>>>>>> a little back to OWASP.
>>>>>>>>>>>>>> However, my instinct and evidence tell me that they both
>>>>>>>>>>>>>> over the line in the use of the OWASP brand to sell product.
>>>>>>>>>>>>>> Now, Jeff makes a good point. We as a board and staff are
>>>>>>>>>>>>>> poor at enforcing brand management policy and it's not fair
>>>>>>>>>>>>>>to single out
>>>>>>>>>>>>>> Contrast, when many other vendors violate the brand, IMO.
>>>>>>>>>>>>>>Just google OWASP
>>>>>>>>>>>>>> and watch the ads fly that use the OWASP name to sell
>>>>>>>>>>>>>> Also, any and every request that was made of Dave to adjust
>>>>>>>>>>>>>> project for the sake of vendor neutrality was taken very
>>>>>>>>>>>>>> Regardless of Daves past intentions, he is clearly trying
>>>>>>>>>>>>>>to do the right
>>>>>>>>>>>>>> thing moving forward.
>>>>>>>>>>>>>> I look to "postels principle" in this situation (this is
>>>>>>>>>>>>>> otherwise known as the "robustness principle" and dates
>>>>>>>>>>>>>>back to the creation
>>>>>>>>>>>>>> of TCP) . This is paraphrased as, "Be liberal in what you
>>>>>>>>>>>>>>take from others
>>>>>>>>>>>>>> but be conservative in what you dish out". So I think it's
>>>>>>>>>>>>>>critical that
>>>>>>>>>>>>>> OWASP and any OWASP resource present itself in a strict
>>>>>>>>>>>>>>vendor neutral way.
>>>>>>>>>>>>>> But unless OWASP wants to be much more "even" in the
>>>>>>>>>>>>>>enforcement of brand
>>>>>>>>>>>>>> policy across the board to all violators, we should be
>>>>>>>>>>>>>>fairly lax in the
>>>>>>>>>>>>>> enforcement of these issues from the outside world.
>>>>>>>>>>>>>> I am trying to be objective here. My trigonometry teacher
>>>>>>>>>>>>>> told me "I'd fail my mother" when I asked him if he would
>>>>>>>>>>>>>>ever fail me (I
>>>>>>>>>>>>>> was an A student). If my mother owned a security company
>>>>>>>>>>>>>>and tried the same
>>>>>>>>>>>>>> stunt, I'd have the same opinions about her actions as well.
>>>>>>>>>>>>>> So what next? Well hello from the other side. I'm going
>>>>>>>>>>>>>>back to
>>>>>>>>>>>>>> listening to Adele's new album where I can sit in my deep
>>>>>>>>>>>>>>feelings and
>>>>>>>>>>>>>> reflect upon what the OWASP foundation has done to enrich
>>>>>>>>>>>>>>my life. I would
>>>>>>>>>>>>>> much rather keep out of this (and any other conflict laden
>>>>>>>>>>>>>>situation at
>>>>>>>>>>>>>> OWASP), but I feel it's my responsibility to speak up.
>>>>>>>>>>>>>> Aloha,
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Jim Manico
>>>>>>>>>>>>>> Global Board Member
>>>>>>>>>>>>>> OWASP Foundation
>>>>>>>>>>>>>> https://www.owasp.org
>>>>>>>>>>>>>> Join me in Rome for AppSecEU 2016!
>>>>>>>>>>>>>> On Nov 26, 2015, at 9:09 PM, Josh Sokol
>>>>>>>>>>>>>><josh.sokol at owasp.org>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> I would be happy to provide an update.
>>>>>>>>>>>>>> Matt Konda and Dave Wichers, the Benchmark Project Leader,
>>>>>>>>>>>>>> a conversation a few weeks back.  To summarize their
>>>>>>>>>>>>>>conversation, Dave
>>>>>>>>>>>>>> acknowledges the currently lack of diversity in his project
>>>>>>>>>>>>>>and it is his
>>>>>>>>>>>>>> sincere desire to drive more people to it to help.  He also
>>>>>>>>>>>>>>acknowledges the
>>>>>>>>>>>>>> issues with Contrast's extreme marketing around the project
>>>>>>>>>>>>>>and feels that
>>>>>>>>>>>>>> it is in everyone's best interests for them to curb it
>>>>>>>>>>>>>>back.  While he does
>>>>>>>>>>>>>> have an ownership stake in Contrast, he works at Aspect and
>>>>>>>>>>>>>>has no control
>>>>>>>>>>>>>> over the marketing messages that they are putting out
>>>>>>>>>>>>>>there.  From the Board
>>>>>>>>>>>>>> perspective, there has been no evidence of any impropriety
>>>>>>>>>>>>>>on Dave's part
>>>>>>>>>>>>>> and it should be our goal to drive more diversity into the
>>>>>>>>>>>>>>project to
>>>>>>>>>>>>>> support Dave.  Dave appears to be sincere in his desires to
>>>>>>>>>>>>>>create a tool
>>>>>>>>>>>>>> where OWASP can tell vendors what we expect from their
>>>>>>>>>>>>>>tools.  If the main
>>>>>>>>>>>>>> issue is that only members of Aspect are working on it,
>>>>>>>>>>>>>>then the best thing
>>>>>>>>>>>>>> that we can do is try to get him some outside assistance.
>>>>>>>>>>>>>>We are also
>>>>>>>>>>>>>> asking that the project be opened up to commits via Git so
>>>>>>>>>>>>>>that outsiders
>>>>>>>>>>>>>> can push commits to it.
>>>>>>>>>>>>>> Josh Sokol and Jeff Williams, the CTO of Contrast, had a
>>>>>>>>>>>>>> conversation a few weeks back.  To summarize their
>>>>>>>>>>>>>>conversation, Jeff
>>>>>>>>>>>>>> believes that the work that Dave is doing on the Benchmark
>>>>>>>>>>>>>>is a game changer
>>>>>>>>>>>>>> in that it gives OWASP the power in dictating what these
>>>>>>>>>>>>>>tools need to be
>>>>>>>>>>>>>> finding.  He wants the Benchmark to be successful and
>>>>>>>>>>>>>>understands that it
>>>>>>>>>>>>>> needs to be diverse in order to be trusted.  He recognizes
>>>>>>>>>>>>>>that Dave is
>>>>>>>>>>>>>> trying to do that and does not want the marketing message
>>>>>>>>>>>>>>from Contrast to
>>>>>>>>>>>>>> interfere with his efforts.  Jeff felt that the "Lab"
>>>>>>>>>>>>>>status granted to
>>>>>>>>>>>>>> Benchmark meant that it was ready for mainstream adoption,
>>>>>>>>>>>>>>that it had 21k
>>>>>>>>>>>>>> tests, and was almost a year old, and didn't see anything
>>>>>>>>>>>>>>wrong with
>>>>>>>>>>>>>> marketing their results, but has agreed to talk to their
>>>>>>>>>>>>>>marketing team to
>>>>>>>>>>>>>> get them to lay off that message for now.  From the Board
>>>>>>>>>>>>>>perspective, we
>>>>>>>>>>>>>> have come to the realization that our brand usage
>>>>>>>>>>>>>>guidelines need an
>>>>>>>>>>>>>> overhaul to clarify what is and is not allowed.  We have
>>>>>>>>>>>>>>made a few
>>>>>>>>>>>>>> proposals and have reached out to Mozilla to gain more
>>>>>>>>>>>>>>insight on their
>>>>>>>>>>>>>> guidelines and even ask for assistance.  Noreen and Paul
>>>>>>>>>>>>>>are taking lead on
>>>>>>>>>>>>>> these efforts.
>>>>>>>>>>>>>> There is a note in the notes that the Board was supposed to
>>>>>>>>>>>>>> follow up with an open letter to the community and
>>>>>>>>>>>>>>companies involved
>>>>>>>>>>>>>> describing our review and actions.  I don't think that has
>>>>>>>>>>>>>>happened so I
>>>>>>>>>>>>>> will remind the person who took on that action item.
>>>>>>>>>>>>>> I'm happy to answer any questions that you may have.
>>>>>>>>>>>>>> ~josh
>>>>>>>>>>>>>> On Thu, Nov 26, 2015 at 11:55 AM, Tobias
>>>>>>>>>>>>>> <tobias.gondrom at owasp.org> wrote:
>>>>>>>>>>>>>>> There have been several conversations on that matter and a
>>>>>>>>>>>>>>> dedicated call. Unfortunately for personal reasons I could
>>>>>>>>>>>>>>>not attend the
>>>>>>>>>>>>>>> last call as it was at 04:00am my local time, but all
>>>>>>>>>>>>>>>other board members
>>>>>>>>>>>>>>> did participate.
>>>>>>>>>>>>>>> Could please one of my fellow board members give an update.
>>>>>>>>>>>>>>> Best, Tobias
>>>>>>>>>>>>>>> On 26/11/15 18:04, Timo Goosen wrote:
>>>>>>>>>>>>>>> I would also like to know the answer to Simon's question.
>>>>>>>>>>>>>>> need to get rid of bad apples in OWASP in my opinion,
>>>>>>>>>>>>>>>there are too many
>>>>>>>>>>>>>>> people just using the OWASP "name" or "brand" to improve
>>>>>>>>>>>>>>>their own financial
>>>>>>>>>>>>>>> situation or career.
>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>> Timo
>>>>>>>>>>>>>>> On Thu, Nov 26, 2015 at 1:13 PM, psiinon
>>>>>>>>>>>>>>><psiinon at gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> Paul, and the rest of the board,
>>>>>>>>>>>>>>>> Its been over 2 months since I raised this issue.
>>>>>>>>>>>>>>>> Whats happening?
>>>>>>>>>>>>>>>> Has the board even discussed it?
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>> On Tue, Oct 20, 2015 at 10:00 PM, Paul Ritchie
>>>>>>>>>>>>>>>> <paul.ritchie at owasp.org> wrote:
>>>>>>>>>>>>>>>>> Eoin, Johanna, All:
>>>>>>>>>>>>>>>>> In an earlier email, Josh Sokol mentioned that he will be
>>>>>>>>>>>>>>>>> speaking in the next day or 2 to their CTO, while at
>>>>>>>>>>>>>>>>>LASCON, as a
>>>>>>>>>>>>>>>>> representative of the OWASP Board.  Following that
>>>>>>>>>>>>>>>>>feedback, the Board has
>>>>>>>>>>>>>>>>> action to take the next steps.
>>>>>>>>>>>>>>>>> Just an FYI that all comments are recognized and action
>>>>>>>>>>>>>>>>> being taken.
>>>>>>>>>>>>>>>>> Paul
>>>>>>>>>>>>>>>>> Best Regards, Paul Ritchie
>>>>>>>>>>>>>>>>> OWASP Executive Director
>>>>>>>>>>>>>>>>> paul.ritchie at owasp.org
>>>>>>>>>>>>>>>>> On Tue, Oct 20, 2015 at 1:54 PM, johanna curiel curiel
>>>>>>>>>>>>>>>>> <johanna.curiel at owasp.org> wrote:
>>>>>>>>>>>>>>>>>> Time for owasp to do a public statement and put a clear
>>>>>>>>>>>>>>>>>> story regarding this abusive behavior of Owasp brand
>>>>>>>>>>>>>>>>>> On Tuesday, October 20, 2015, Eoin Keary
>>>>>>>>>>>>>>>>>> <eoin.keary at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>> Folks,
>>>>>>>>>>>>>>>>>>> The project should be immediately shelved it's simply
>>>>>>>>>>>>>>>>>>> form.
>>>>>>>>>>>>>>>>>>> This is damaging to OWASP, the industry and exactly
>>>>>>>>>>>>>>>>>>> OWASP is not about.
>>>>>>>>>>>>>>>>>>> There is a clear conflict of interest and distinct
>>>>>>>>>>>>>>>>>>>lack of
>>>>>>>>>>>>>>>>>>> science behind the claims made by Contrast.
>>>>>>>>>>>>>>>>>>> Eoin Keary
>>>>>>>>>>>>>>>>>>> OWASP Volunteer
>>>>>>>>>>>>>>>>>>> @eoinkeary
>>>>>>>>>>>>>>>>>>> On 7 Oct 2015, at 3:53 p.m., johanna curiel curiel
>>>>>>>>>>>>>>>>>>> <johanna.curiel at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>> At the moment we did the project review, we observed
>>>>>>>>>>>>>>>>>>> the project did not have enough testing to be
>>>>>>>>>>>>>>>>>>>considered in any form as
>>>>>>>>>>>>>>>>>>> 'ready'  for benchmarking, neither that it had yet the
>>>>>>>>>>>>>>>>>>>community adoption,
>>>>>>>>>>>>>>>>>>> however technically speaking as it has been classified
>>>>>>>>>>>>>>>>>>>by the leaders, the
>>>>>>>>>>>>>>>>>>> project is at the beta stage.
>>>>>>>>>>>>>>>>>>> Indeed , Dave had the push to have the project reviewed
>>>>>>>>>>>>>>>>>>> but it was never clear that later on the project was
>>>>>>>>>>>>>>>>>>>going to be advertisied
>>>>>>>>>>>>>>>>>>> this way. That all happend after the presentation at
>>>>>>>>>>>>>>>>>>> I had my concerns regarding how sensitive is the
>>>>>>>>>>>>>>>>>>> of the project ,but I think we should allow project
>>>>>>>>>>>>>>>>>>>leaders to develop their
>>>>>>>>>>>>>>>>>>> communication strategy even if this has conflict of
>>>>>>>>>>>>>>>>>>>interest. It all depends
>>>>>>>>>>>>>>>>>>> how they behave and how they manage this.
>>>>>>>>>>>>>>>>>>> On Tuesday, October 6, 2015, Michael Coates
>>>>>>>>>>>>>>>>>>> <michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>>> It's not really that formal to add to the agenda,
>>>>>>>>>>>>>>>>>>>>just a
>>>>>>>>>>>>>>>>>>>> wiki that we add in the text.
>>>>>>>>>>>>>>>>>>>> I think you can safely assume it will get the
>>>>>>>>>>>>>>>>>>>> discussion.
>>>>>>>>>>>>>>>>>>>> On Oct 6, 2015, at 7:16 AM, psiinon
>>>>>>>>>>>>>>>>>>>><psiinon at gmail.com>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Really?? Its not on the agenda yet for the next
>>>>>>>>>>>>>>>>>>>> How does it get added to the agenda?
>>>>>>>>>>>>>>>>>>>> And that was a formal request if that makes any
>>>>>>>>>>>>>>>>>>>> difference :)
>>>>>>>>>>>>>>>>>>>> I'm all in favour of getting the facts straight before
>>>>>>>>>>>>>>>>>>>> any actions are taken, hence my request for an
>>>>>>>>>>>>>>>>>>>>'ethical review' or whatever
>>>>>>>>>>>>>>>>>>>> it should be called.
>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 3:07 PM, Michael Coates
>>>>>>>>>>>>>>>>>>>> <michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>>>> First step is to get all of our information straight so
>>>>>>>>>>>>>>>>>>>>> we're clear on where things are at.
>>>>>>>>>>>>>>>>>>>>> This was not on the board agenda last meeting and is
>>>>>>>>>>>>>>>>>>>>> also not on the next agenda as of yet (of course it could
always be added if
>>>>>>>>>>>>>>>>>>>>> needed).
>>>>>>>>>>>>>>>>>>>>> We are aware that people have raised questions though.
>>>>>>>>>>>>>>>>>>>>> I'm hoping we can get a clear understanding of all the
facts and then
>>>>>>>>>>>>>>>>>>>>> discuss if changes are needed.
>>>>>>>>>>>>>>>>>>>>> On Oct 6, 2015, at 1:52 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> Hey Michael,
>>>>>>>>>>>>>>>>>>>>> Is the board going to take any action?
>>>>>>>>>>>>>>>>>>>>> Were there any discussions about this controversy in the
>>>>>>>>>>>>>>>>>>>>> board meeting at AppSec USA?
>>>>>>>>>>>>>>>>>>>>> If not will it be on the agenda for the meeting on
>>>>>>>>>>>>>>>>>>>>> October 14th?
>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 8:25 AM, Michael Coates
>>>>>>>>>>>>>>>>>>>>> <michael.coates at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>>>>>>> I posted the below message earlier today. At this point
>>>>>>>>>>>>>>>>>>>>>> my goal is to just gain clarity over the current reality
and ideally drive
>>>>>>>>>>>>>>>>>>>>>> to a shared state of success. This message doesn't seem
to be reflected in
>>>>>>>>>>>>>>>>>>>>>> the list yet. It could be because my membership hasn't
been approved or
>>>>>>>>>>>>>>>>>>>>>> because of mail list delays (I miss Google groups). But I
think these
>>>>>>>>>>>>>>>>>>>>>> questions will start the conversation.
>>>>>>>>>>>>>>>>>>>>>> (This was just me asking questions as a curious Owasp
>>>>>>>>>>>>>>>>>>>>>> member, not any action on behalf of the board)
>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message:
>>>>>>>>>>>>>>>>>>>>>> From: Michael Coates <michael.coates at owasp.org>
>>>>>>>>>>>>>>>>>>>>>> Date: October 5, 2015 at 6:20:23 PM PDT
>>>>>>>>>>>>>>>>>>>>>> To: owasp-benchmark-project at lists.owasp.org
>>>>>>>>>>>>>>>>>>>>>> Subject: Project Questions
>>>>>>>>>>>>>>>>>>>>>> OWASP Benchmark List,
>>>>>>>>>>>>>>>>>>>>>> I've heard more about this project and am excited about
>>>>>>>>>>>>>>>>>>>>>> the idea of an independent perspective of tool
performance. I'm trying to
>>>>>>>>>>>>>>>>>>>>>> understand a few things to better respond to questions
from those in the
>>>>>>>>>>>>>>>>>>>>>> security & OWASP community.
>>>>>>>>>>>>>>>>>>>>>> In my mind there are two big areas for consideration in
>>>>>>>>>>>>>>>>>>>>>> a benchmark process.
>>>>>>>>>>>>>>>>>>>>>> 1. Are the benchmarks testing the right areas?
>>>>>>>>>>>>>>>>>>>>>> 2. Is the process for creating the benchmark objective
>>>>>>>>>>>>>>>>>>>>>> & free from conflicts of interest.
>>>>>>>>>>>>>>>>>>>>>> I think as a group OWASP is the right body to align on
>>>>>>>>>>>>>>>>>>>>>> #1.
>>>>>>>>>>>>>>>>>>>>>> I'd like to ask for some clarifications on item #2. I
>>>>>>>>>>>>>>>>>>>>>> think it's important to avoid actual conflict of interest
and also the
>>>>>>>>>>>>>>>>>>>>>> appearance of conflict of interest. The former is obvious
why we mustn't
>>>>>>>>>>>>>>>>>>>>>> have that, the latter is critical so others have faith in
the tool, process
>>>>>>>>>>>>>>>>>>>>>> and outputs of the process when viewing or hearing about
the project.
>>>>>>>>>>>>>>>>>>>>>> 1) Can we clarify whether other individuals have
>>>>>>>>>>>>>>>>>>>>>> submitted meaningful code to the project?
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> Nearly all the code commits have come from 1 person
>>>>>>>>>>>>>>>>>>>>>> (project lead).
>>>>>>>>>>>>>>>>>>>>>> https://github.com/OWASP/Benchmark/graphs/contributors
>>>>>>>>>>>>>>>>>>>>>> 2) Can we clarify the contributions of others and their
>>>>>>>>>>>>>>>>>>>>>> represented organizations?
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> The acknowledgements tab listed two developers (Juan
>>>>>>>>>>>>>>>>>>>>>> Gama & Nick Sanidas) both who work at the same company as
the project lead.
>>>>>>>>>>>>>>>>>>>>>> It seems other people have submitted some small amounts
of material, but
>>>>>>>>>>>>>>>>>>>>>> overall it seems all development has come from the same
>>>>>>>>>>>>>>>>>>>>>> 3) Can we clarify in what ways we've mitigated the
>>>>>>>>>>>>>>>>>>>>>> potential conflict of interest and also the appearance of
a conflict of
>>>>>>>>>>>>>>>>>>>>>> interest? This seems like the largest blocker for wide
spread acceptance of
>>>>>>>>>>>>>>>>>>>>>> this project and the biggest risk.
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> The project lead and both of the project developers
>>>>>>>>>>>>>>>>>>>>>> works for a company with very close ties to one of the
companies that is
>>>>>>>>>>>>>>>>>>>>>> evaluated by this project. Further, it appears the
company is performing
>>>>>>>>>>>>>>>>>>>>>> very well on the project tests.
>>>>>>>>>>>>>>>>>>>>>> 4) If we are going to list tool vendors then I'd
>>>>>>>>>>>>>>>>>>>>>> recommend listing multiple vendors for each category.
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> The tools page only lists 1 IAST tool. Since this is
>>>>>>>>>>>>>>>>>>>>>> the point of the potential conflict of interest it is
important to list
>>>>>>>>>>>>>>>>>>>>>> numerous IAST tools.
>>>>>>>>>>>>>>>>>>>>>> 5) Diverse body with multiple points of view
>>>>>>>>>>>>>>>>>>>>>> Observation:
>>>>>>>>>>>>>>>>>>>>>> There is no indication that multiple stakeholders are
>>>>>>>>>>>>>>>>>>>>>> present to review and decide on the future of this
project. If they exist, a
>>>>>>>>>>>>>>>>>>>>>> new section should be added to the project page to raise
awareness. If they
>>>>>>>>>>>>>>>>>>>>>> don't exist, we should reevaluate how we are obtaining an
independent view
>>>>>>>>>>>>>>>>>>>>>> of the testing process.
>>>>>>>>>>>>>>>>>>>>>> Again, I think the idea of the project is great. From
>>>>>>>>>>>>>>>>>>>>>> my perspective clarifying these questions will help
ensure the project is
>>>>>>>>>>>>>>>>>>>>>> not only objective, but also perceived as objective from
someone reviewing
>>>>>>>>>>>>>>>>>>>>>> the material. Ultimately this will contribute to the
success and growth of
>>>>>>>>>>>>>>>>>>>>>> the project.
>>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Michael Coates
>>>>>>>>>>>>>>>>>>>>>> On Oct 2, 2015, at 1:31 AM, psiinon <psiinon at gmail.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> OK, based on the concerns raised so far I think the
>>>>>>>>>>>>>>>>>>>>>> board should initiate a review of the OWASP Benchmark
>>>>>>>>>>>>>>>>>>>>>> I'm not raising a formal complaint against it, I'm just
>>>>>>>>>>>>>>>>>>>>>> requesting a review.
>>>>>>>>>>>>>>>>>>>>>> And I dont think it needs a 'standard' project review -
>>>>>>>>>>>>>>>>>>>>>> Johanna has already done a very good job of this.
>>>>>>>>>>>>>>>>>>>>>> Not sure what sort of review you'd call it, I'll leave
>>>>>>>>>>>>>>>>>>>>>> the naming to others :)
>>>>>>>>>>>>>>>>>>>>>> I'm concerned that we have an OWASP project lead by a
>>>>>>>>>>>>>>>>>>>>>> company who has a clear commercial stake in the results.
>>>>>>>>>>>>>>>>>>>>>> Bringing more companies on board will help, but I'm
>>>>>>>>>>>>>>>>>>>>>> still not sure that alone will make it independent
>>>>>>>>>>>>>>>>>>>>>> Commercial companies can afford to dedicate staff to
>>>>>>>>>>>>>>>>>>>>>> improving Benchmark so that their products look better.
>>>>>>>>>>>>>>>>>>>>>> Open source projects just cant do that, so we are at a
>>>>>>>>>>>>>>>>>>>>>> distinct disadvantage.
>>>>>>>>>>>>>>>>>>>>>> Should we allow a commercially driven OWASP project
>>>>>>>>>>>>>>>>>>>>>> who's aim could be seen be to promote commercial
>>>>>>>>>>>>>>>>>>>>>> If so, what sort of checks and balances does it need?
>>>>>>>>>>>>>>>>>>>>>> Those are the sort of questions I'd like an independent
>>>>>>>>>>>>>>>>>>>>>> review to look at.
>>>>>>>>>>>>>>>>>>>>>> I do think there are some immediate steps that could be
>>>>>>>>>>>>>>>>>>>>>> taken:
>>>>>>>>>>>>>>>>>>>>>> I'd like to see the Benchmark project page clearly
>>>>>>>>>>>>>>>>>>>>>> state thats its at a very early stage and that the
results are _not_ yet
>>>>>>>>>>>>>>>>>>>>>> suitable for use in commercial literature.
>>>>>>>>>>>>>>>>>>>>>> I'd also like the main companies developing Benchmark
>>>>>>>>>>>>>>>>>>>>>> to be clearly stated on the main page. If and when other
companies get
>>>>>>>>>>>>>>>>>>>>>> involved then this would actually help the project's
claim of vendor
>>>>>>>>>>>>>>>>>>>>>> independence.
>>>>>>>>>>>>>>>>>>>>>> And I'd love to see a respected co-leader added to the
>>>>>>>>>>>>>>>>>>>>>> project who is not associated with any commercial or open
source security
>>>>>>>>>>>>>>>>>>>>>> tools:)
>>>>>>>>>>>>>>>>>>>>>> And we should carry on discussing the project on this
>>>>>>>>>>>>>>>>>>>>>> list - I think such discussions are very healthy, and I'd
love to see this
>>>>>>>>>>>>>>>>>>>>>> project mature to a state where it can be a trusted,
independent and valued
>>>>>>>>>>>>>>>>>>>>>> resource.
>>>>>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>>>>>>>>> On Thu, Oct 1, 2015 at 7:59 PM, Tobias
>>>>>>>>>>>>>>>>>>>>>> <tobias.gondrom at owasp.org> wrote:
>>>>>>>>>>>>>>>>>>>>>>> @Simon:
>>>>>>>>>>>>>>>>>>>>>>> yes, the leaders list is the place for your
>>>>>>>>>>>>>>>>>>>>>>> discussions for project and chapter leaders
>>>>>>>>>>>>>>>>>>>>>>> @Timo: I like your framing of "Don't ask what OWASP
>>>>>>>>>>>>>>>>>>>>>>> can do for me, ask what I can do for OWASP."
>>>>>>>>>>>>>>>>>>>>>>> That should and is indeed the spirit of OWASP:-)
>>>>>>>>>>>>>>>>>>>>>>> Best regards, Tobias
>>>>>>>>>>>>>>>>>>>>>>> On 30/09/15 09:42, Timo Goosen wrote:
>>>>> ...
>>>>> [Message clipped]
>>>>> _______________________________________________
>>>>> Owasp-board mailing list
>>>>> Owasp-board at lists.owasp.org
>>>>> https://lists.owasp.org/mailman/listinfo/owasp-board
>>>> --
>>>> OWASP ZAP Project leader
>>>> _______________________________________________
>>>> OWASP-Leaders mailing list
>>>> OWASP-Leaders at lists.owasp.org
>>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>> --
>> OWASP ZAP Project leader
>> _______________________________________________
>> OWASP-Leaders mailing list
>> OWASP-Leaders at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>OWASP-Leaders mailing list
>OWASP-Leaders at lists.owasp.org

OWASP-Leaders mailing list
OWASP-Leaders at lists.owasp.org

OWASP-Leaders mailing list
OWASP-Leaders at lists.owasp.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151201/a5b4cee7/attachment-0001.html>

More information about the OWASP-Leaders mailing list