Of course it's hard! If it wasn't we'd have tons of "benchmark" programs. Jeff, I am absolutely certain that you've done a great, objective work with the benchmark project. <div>I strongly believe that owasp as a vendor neutral organization should not make any kind of judgement or provide metrics for comparing vendors. If I was a competing vendor I'd be seriously considering to withdraw my support for owasp eg as a corporate sponsor.</div><div><br></div><div><br><br>On Wednesday, 2 December 2015, Jeff Williams <<a href="mailto:jeff.williams@owasp.org">jeff.williams@owasp.org</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
    <div style="padding-left:20px;padding-right:20px;padding-bottom:8px"><div>You've mischaracterized the situation here. Gartner has no prayer of producing something like this.  It's not like it's hard to verify that the test cases are fair and reasonable.</div><div><br><div>--Jeff<br></div></div></div>
    <div class="gmail_quote">_____________________________<br>From: Tony Turner <<a dir="ltr" href="javascript:_e(%7B%7D,'cvml','tony.turner@owasp.org');" target="_blank">tony.turner@owasp.org</a>><br>Sent: Tuesday, December 1, 2015 3:05 PM<br>Subject: Re: [Owasp-leaders] [Owasp-board] OWASP Benchmark project - potential conflict of interest<br>To: Konstantinos Papapanagiotou <<a dir="ltr" href="javascript:_e(%7B%7D,'cvml','konstantinos@owasp.org');" target="_blank">konstantinos@owasp.org</a>><br>Cc: OWASP Foundation Board List <<a dir="ltr" href="javascript:_e(%7B%7D,'cvml','owasp-board@lists.owasp.org');" target="_blank">owasp-board@lists.owasp.org</a>>,  <<a dir="ltr" href="javascript:_e(%7B%7D,'cvml','owasp-leaders@lists.owasp.org');" target="_blank">owasp-leaders@lists.owasp.org</a>><br><br><br>    <div dir="ltr">   I would oppose setting a precedent that states an OWASP project cannot perform such evaluations. We just need to establish some common sense guidelines here, like not allowing project leadership to biased by employer/partnership relationships or requiring a multi-leader PM structure. What I do think makes sense, is for OWASP the organization to not rank products. Its a very fine distinction to be sure.   <div>    <br>   </div>   <div>    I do NOT have an issue with the benchmark project's core purpose. I DO have an issue with it being led by Contrast (or any SAST/DAST/IAST vendor). I DO have an issue with their abuse of OWASP brand for marketing purposes.   </div>   <div>    <br>   </div>  </div>  <div class="gmail_extra">   <br>   <div class="gmail_quote">    On Tue, Dec 1, 2015 at 1:57 PM, Konstantinos Papapanagiotou     <span dir="ltr"><<a href="javascript:_e(%7B%7D,'cvml','Konstantinos@owasp.org');" target="_blank">Konstantinos@owasp.org</a>></span> wrote:    <br>    <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">     <div dir="ltr">      <div>       <div>        <div>         <div>          My 2 cents on this:          <br>         </div>         <br>        </div>I am quite (and unpleasantly) surprised that we need several threads, a resignation of a volunteer and numerous comments on social media to debate and reach a conclusion on this issue. From where I come from the situation has been quite clear from the beginning. Imagine an antivirus vendor building an av assessment tool that -what a coincidence- ranks them as their product as the best av, donating it to VirusBulletin and then saying that VirusBulletin has concluded that their av is the best. Personally it makes me feel stupid; Contrast's marketing approach undermines my intelligence.        <br>        <br>       </div>The glass is broken. In my opinion we need an official statement saying the "the benchmark" is no longer an OWASP project and that OWASP does not endorse any vendor. In my opinion OWASP should not be in the business of testing and ranking different commercial products. We are not Gartner and we should not become Gartner. I honestly can't see any other way of fixing this apart from removing the project from the OWASP inventory.       <br>       <br>      </div>Kostas      <br>     </div>     <div>      <div>       <div class="gmail_extra">        <br>        <div class="gmail_quote">         On Mon, Nov 30, 2015 at 7:17 PM, psiinon          <span dir="ltr"><<a href="javascript:_e(%7B%7D,'cvml','psiinon@gmail.com');" target="_blank">psiinon@gmail.com</a>></span> wrote:         <br>         <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">          <div dir="ltr">           I'd like to start by saying that I actually _like_ the Benchmark project.           <br>Myself and other ZAP developers have made some contributions to it, and we have used (and will continue to use) it to make ZAP better.           <br>I think these sort of testing applications are very valuable to all security tools, and I'd like to thank Dave and his team for the significant amount of effort involved in developing and open sourcing it.           <br>           <br>But I dont think it should be an OWASP project.           <br>I do not think that a vendor led project can ever objectively evaluate competing commercial and open source projects.           <br>I do not think that just saying 'pull requests welcomed' makes a project vendor neutral.           <br>I do not think that a project as mired in controversy as the Benchmark project can ever recover to become truly independent.           <br>           <br>I am very disappointed in the Boards handling of this affair.           <br>           <br>Ideally I'd like Dave to understand how much damage this project has done and to withdraw it as an OWASP project, while still maintaining it as a very valuable vendor led open source resource.           <br>           <br>Failing that I really hope that the Board comes to its senses and ejects the Benchmark project before even more damage is done.           <br>At the _very_ least it should flag the project as being 'in dispute' (as Kevin suggested) while a more detailed evaluation is performed.           <br>           <br>However I'm rapidly loosing loosing faith that the Board will do the right thing and protect OWASP's image in the way that they should have already done.           <br>Members - please make your voices heard before more people and projects leave OWASP.           <br>           <br>Simon           <br>          </div>          <div class="gmail_extra">           <br>           <div class="gmail_quote">            On Sat, Nov 28, 2015 at 5:14 AM, Jim Manico             <span dir="ltr"><<a href="javascript:_e(%7B%7D,'cvml','jim.manico@owasp.org');" target="_blank">jim.manico@owasp.org</a>></span> wrote:            <br>            <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">             <div dir="auto">              <div>               WAFEC does not "do vendor assessment"; they define a comprehensive standard built by many vendors and let the community use that standard to measure tools on their own. Just a FYI, I was involved in the early version of this project. (Things may have changed since my involvement, I'm sure Tony has more details here)              </div>              <div>               <br>              </div>              <div>               Johanna's comments on this issue lead me to believe that the damage done to both OWASP and DHS is even more destructive that I thought. It saddens me to see this level of abuse just to sell product.              </div>              <span>               <div>                <br>                <div>                 --                </div>                <div>                 Jim Manico                </div>                <div>                 <div>                  <div style="word-wrap:break-word">                   <div>                    <span style="background-color:rgba(255,255,255,0)">Global Board Member</span>                   </div>                   <span style="background-color:rgba(255,255,255,0)">OWASP Foundation</span>                   <div>                    <a href="https://www.owasp.org/" style="background-color:rgba(255,255,255,0)" target="_blank"><font color="#000000">https://www.owasp.org</font></a>                   </div>                  </div>                 </div>                 <div>                  <span style="background-color:rgba(255,255,255,0)">Join me in Rome for AppSecEU 2016!</span>                 </div>                </div>               </div></span>              <div>               <div>                <div>                 <br>On Nov 28, 2015, at 2:40 AM, Josh Sokol <                 <a href="javascript:_e(%7B%7D,'cvml','josh.sokol@owasp.org');" target="_blank">josh.sokol@owasp.org</a>> wrote:                 <br>                 <br>                </div>                <blockquote>                 <div>                  <p dir="ltr">One of the ideas that Andrew proposed was actually approaching WAFEC to learn more about how they do vendor assessment in a neutral way.  It's great to hear that we have a resource here already that we can leverage.  I wasn't aware of your affiliation. </p>                   <p dir="ltr">~josh</p>                   <div class="gmail_quote">                   On Nov 27, 2015 2:47 PM, "Tony Turner" <                   <a href="javascript:_e(%7B%7D,'cvml','tony.turner@owasp.org');" target="_blank">tony.turner@owasp.org</a>> wrote:                   <br>                   <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                    <p dir="ltr">I sincerely hope so. That's not the impression I got from others comments. Personally I haven't used the tool at all, but as I'm the project lead for another product evaluation project (WAFEC) I'm very sensitive to the need of collaboration with many different vendors. There really has to be a very high level (almost paranoid level) transparency with how vendors are approached, worked with, how requirements for evaluation are defined, and how metrics are derived.  </p>                     <p dir="ltr">It appears the project team is attempting to address these last 2 somewhat but I'd like to see more specifics, and the lack of information on how they are addressing vendor communication, participation and transparency seems a bit concerning. Lastly, it is my opinion that project leadership should not belong to anyone working for or with a partnership/ownership stake for any vendor being evaluated. I think this is a flawed model and should transition to a vendor neutral party. </p>                     <div class="gmail_quote">                     On Nov 27, 2015 3:16 PM, "Josh Sokol" <                     <a href="javascript:_e(%7B%7D,'cvml','josh.sokol@owasp.org');" target="_blank">josh.sokol@owasp.org</a>> wrote:                     <br>                     <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                      <div dir="ltr">                       <div>                        I don't know what qualifies as "significant" in your mind, but my understanding is that there have been contributions from other vendors:                        <br>                        <br>                        <a href="https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements" target="_blank">https://www.owasp.org/index.php/Benchmark#tab=Acknowledgements</a>                        <br>                        <br>                       </div>                       <div>                        Still, Dave would like more, but he can't force them to help.                        <br>                       </div>                       <div>                        <br>                       </div>~josh                       <br>                      </div>                      <div class="gmail_extra">                       <br>                       <div class="gmail_quote">                        On Fri, Nov 27, 2015 at 1:45 PM, Tony Turner                         <span dir="ltr"><<a href="javascript:_e(%7B%7D,'cvml','tony.turner@owasp.org');" target="_blank">tony.turner@owasp.org</a>></span> wrote:                        <br>                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                         <p dir="ltr">While I can appreciate that they started with Contrast, if there hasn't been significant effort to include other vendors it's a worthless benchmark. It's easy to state you haven't gotten support from other vendors and that's fine, but until you do there's really nothing to release. Why was it ever upgraded? Talking about the results without an accurate comparative analysis is akin to snake oil. </p>                          <div class="gmail_quote">                          On Nov 27, 2015 1:49 PM, "Josh Sokol" <                          <a href="javascript:_e(%7B%7D,'cvml','josh.sokol@owasp.org');" target="_blank">josh.sokol@owasp.org</a>> wrote:                          <br>                          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                           <div dir="ltr">                            <div>                             <div>                              Thank you for the links to those articles.  The first one discusses the strengths and weaknesses of the different methods of evaluating for application vulnerabilities.  The section on the Benchmark seems wholly appropriate to me.  That seems like an excellent description of what the project is designed to do.  I see some metrics in there about which tools are more effective on which types of vulnerabilities, but I don't see him straight up saying "The OWASP Benchmark proves that Contrast is better".  This seems like statements made based on some level of testing and research.  Honestly, I don't see any OWASP brand abuse in that article.  Whether it's in good taste or not at this stage in the project is certainly debatable, but if you look at the brand usage guidelines (                              <a href="https://www.owasp.org/index.php/Marketing/Resources#tab=BRAND_GUIDELINES" target="_blank">https://www.owasp.org/index.php/Marketing/Resources#tab=BRAND_GUIDELINES</a>), I don't see any violations.  We need to govern to policy here which is why Paul and Noreen are evaluating changes to the guidelines and our enforcement policies to make abuse more difficult.                              <br>                              <br>                             </div>The second article is a competing vendor's reaction to the first.  He makes some good points about the issues with Benchmark, but he also says that he hopes that it will be improved over time, and Dave has committed to that.  What I don't see is the vendor saying "...and Veracode has committed resources to help make the Benchmark more accurate across all tool sets".  The Benchmark page is pretty clear that it does it's best to provide a benchmark without working exactly like a real-world application.  Maybe some more disclaimer text about where the project is at today would be in order to validate some of Chris' concerns, but I hardly see this as "brand abuse" or a reason to demote the project.                             <br>                             <br>                            </div>                            <div>                             Please consider that I have spoken with both Dave and Jeff on this topic and read much of the discussions around it before formulating my opinion.  I doubt that you have done the same so I'm not sure how you can claim that you have researched the issues and all parties involved when you haven't even spoken with the two people whom you are accusing of impropriety.  I have no bias here.  I am simply speaking with the individuals involved, looking at the currently OWASP policies and guidelines, and helping to determine our next steps.                               <br>                            </div>                            <div>                             <br>                            </div>~josh                            <br>                           </div>                           <div>                            <br>                            <div>                             On Fri, Nov 27, 2015 at 12:22 PM, johanna curiel curiel                              <span dir="ltr"><<a>johanna.curiel@owasp.org</a>></span> wrote:                             <br>                             <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                              <div dir="ltr">                               <span style="font-size:13px">>>While I agree with you that there has been some brand abuse, it was abuse by Contrast (specifically their marketing department), and not by "these gentlemen" as  you state. </span>                               <div>                                <br>                               </div>                               <div>                                Really? ..'some brand abuse'..this is more than brand abuse                                <br>                                <div>                                 <span style="font-size:13px"><br></span>                                </div>                                <div>                                 <span style="font-size:13px">Josh , please read also the article written by Jeff</span>                                </div>                               </div>                               <div>                                <a href="http://www.darkreading.com/vulnerabilities---threats/why-its-insane-to-trust-static-analysis/a/d-id/1322274" target="_blank">http://www.darkreading.com/vulnerabilities---threats/why-its-insane-to-trust-static-analysis/a/d-id/1322274</a>?                                <span style="font-size:13px"><br></span>                               </div>                               <div>                                <span style="font-size:13px"><br></span>                               </div>                               <div>                                And Veracode's reaction including others in Twitter                               </div>                               <div>                                <a href="https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet" target="_blank">https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet</a>                                <br>                               </div>                               <div>                                <br>                               </div>                               <div>                                My strong advice is to research the issues and all the parties involved before making statements                               </div>                               <div>                                <br>                               </div>                               <div>                                <br>                               </div>                               <div>                                <br>                               </div>                              </div>                              <div>                               <br>                               <div>                                On Fri, Nov 27, 2015 at 2:07 PM, Josh Sokol                                 <span dir="ltr"><<a>josh.sokol@owasp.org</a>></span> wrote:                                <br>                                <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                                 <div dir="ltr">                                  <div>                                   <div>                                    Jim,                                    <br>                                    <br>                                   </div>A concern was expressed to the Board and, frankly, I am insulted by you saying that this was "brushed under the rug".  The Board delegated Matt to talk with Dave and they had a lengthy conversation on the subject.  The Board delegated me to talk with Jeff and we had a lengthy conversation on the subject.  If you do not trust in our abilities to read people, ask the right questions, and provide honest feedback about our conversations, then that's a bigger issue that we should take offline.  After our conversations, we took the time to call a special two-hour session of the Board in order to discuss this subject (and only this subject).  We spoke about all facets of the issue at hand, about the challenges and possible solutions, and concluded on some very concrete next steps.                                     <br>                                   <br>While I agree with you that there has been some brand abuse, it was abuse by Contrast (specifically their marketing department), and not by "these gentlemen" as  you state.  Unless you can point to some sort of evidence showing that Jeff and/or Dave first-hand abused the brand, then I believe that you are speaking with your heart instead of with your head.  I appreciate your passion, but I label this as conspiracy theory because without evidence to support your claims, I cannot accept it as anything other.                                   <span><font color="#888888"><br><br></font></span>                                  </div>                                  <span><font color="#888888">~josh <br></font></span>                                 </div>                                 <div>                                  <div>                                   <div>                                    <br>                                    <div>                                     On Fri, Nov 27, 2015 at 11:39 AM, Jim Manico                                      <span dir="ltr"><<a>jim.manico@owasp.org</a>></span> wrote:                                     <br>                                     <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                                      <div dir="auto">                                       <div>                                        Josh,                                       </div>                                       <div>                                        <br>                                       </div>                                       <div>                                        I stand by my comments and perspective, but I'm disheartened that you consider my presentation of facts (and the concerns of many active members of our community) as a "conspiracy theory".                                       </div>                                       <div>                                        <br>                                       </div>                                       <div>                                        In my experience, these kind of comments border on insults and only cause folks to harden their opinions.                                        <br>                                        <br>Once again I feel these gentlemen got away with a kind of brand abuse that is very hurtful to the OWASP community but I am at a loss as to how handle or prevent these kinds of mishaps - especially when board members like yourself seem willing to - from what I see - brush it under the rug.                                       </div>                                       <span>                                        <div>                                         <br>                                        </div>                                        <div>                                         <div>                                          --                                         </div>                                         <div>                                          Jim Manico                                         </div>                                         <div>                                          <div>                                           <div style="word-wrap:break-word">                                            <div>                                             <span style="background-color:rgba(255,255,255,0)">Global Board Member</span>                                            </div>                                            <span style="background-color:rgba(255,255,255,0)">OWASP Foundation</span>                                            <div>                                             <a href="https://www.owasp.org/" style="background-color:rgba(255,255,255,0)" target="_blank"><font color="#000000">https://www.owasp.org</font></a>                                            </div>                                           </div>                                          </div>                                          <div>                                           <span style="background-color:rgba(255,255,255,0)">Join me in Rome for AppSecEU 2016!</span>                                          </div>                                         </div>                                        </div></span>                                       <div>                                        <div>                                         <div>                                          <br>On Nov 27, 2015, at 7:23 PM, Josh Sokol <                                          <a>josh.sokol@owasp.org</a>> wrote:                                          <br>                                          <br>                                         </div>                                         <blockquote>                                          <div>                                           <div dir="ltr">                                            <div>                                             <div>                                              <div>                                               <div>                                                <div>                                                 <div>                                                  Admittedly, this was my gut reaction at first as well.  I began linking all of these companies, people, and projects together in my mind (there are some loose links there) and painted a big conspiracy picture similar to what Jim and Dinis have stated.  But, after speaking directly with Jeff, and hearing about the conversation that Dave and Matt had, I've changed my mind.                                                    <br>                                                  <br>                                                 </div>I think it begins with the project itself.  If you aren't sold on the idea of the Benchmark, then you'll never be able to get to the same place.  My original line of thinking was that it was just a bar for vendors to compare their tools against eachother, but that's a bit myopic.  We are in an industry where things evolve very quickly.  As a customer of these tools, I know firsthand that something that a tool does today may not be the case a week from now.  Likewise, new features are being added daily and I need a point-in-time metric to be able to gauge continual effectiveness.  Cool, right?  But not a game changer.  The game changer part comes when you realize that by developing and evolving the tests that go into the Benchmark, we are moving the bar higher and higher.  We (OWASP) are effectively setting the standard by which these tools will be compared.  A tool that receives a lower score on the Benchmark today knows exactly what they need to work on in order to pass that test tomorrow and we already have examples of tools that have made improvements because of their Benchmark score (Ask Simon about ZAP's experience with the Benchmark).  I don't think that anyone can argue that the Benchmark project isn't being effective when OWASP's own tools are being driven forward as a result of using it.                                                 <br>                                                 <br>                                                </div>But, but, but, Dave and Jeff own Aspect and have stock in Contrast and Jeff is the Contrast CTO and Contrast got good scores so it's a conspiracy right?  Is there some code that allows Contrast to use the Benchmark?  Absolutely.  Can you really blame Dave for starting his testing on the effectiveness of the Benchmark with a tool that he owned and is familiar with?  If I were going to start a similar project, there's no question in my mind that I would begin my testing with the tools that I have available to me.  That said, is there code that allows other tools to use the Benchmark?  Absolutely.                                                  <br>                                                <br>                                               </div>Regarding "Dave has a history of breaching his duty to be vendor neutral", while I cannot comment on his past actions, I can judge what we've seen recently.  Matt saw a presentation from Dave on the Benchmark at a conference in Chicago.  He said that he felt that the message was appropriate and while IAST tools were mentioned as receiving higher scores, it wasn't a "Contrast is the best" type of message, more of a generality.  I saw a very similar (if not the same) talk by Jeff at LASCON 2015 and the message was exactly the same.  I watched the talk expecting some sort of impropriety, but found none.  So, perhaps Dave has abused some privilege granted to him in the past, but what I've seen from him at this point, with respect to the Benchmark, has been appropriate.                                               <br>                                               <br>                                              </div>You have a very good point with respect to the Contrast marketing message around the Benchmark.  It's been completely absurd, over the top, and, in my personal opinion, intolerable.  In fact, I experienced the same thing that you talked about with them at LASCON 2015 where they stood in front of the door of the room Jeff was speaking in and scanned attendees as they went into the talk.  I agree that these types of aggressive marketing tactics cannot be tolerated at OWASP.  In addition, we have seen several marketing messages from them effectively implying that OWASP endorses Contrast.  Clearly this is not OK.  I've spoken with Jeff about it and we agreed that it is not in the Benchmark's best interest to have this aggressive Contrast marketing around it at such an early stage.  He has said that he is not responsible for Contrast's marketing team, but that he would speak with the people who are.  I haven't seen a single message from them since so I'm guessing that he's made good on this promise.  While that's an excellent start, OWASP's takeaway here should be that we need to do a better job with our brand usage guidelines both in terms of the wording and enforcement.  There are many other companies out there that use the OWASP brand and I think that we agree that selective enforcement against Contrast is not the right answer.  Paul and Noreen are actively working on this.  Either way, I think that implying that activities from a vendor's marketing department means that the project is not objective is not inappropriate.  If we feel that the project is not objective, then separate measures need to be taken to drive contribution diversity into it.  That I absolutely agree with and the message from Dave was that he would love to have more contributors to his project.  But, seeing as we cannot force people to work on it, this becomes a matter of "put up or shut up".  The same goes for the experts that you said reviewed the code.  If they feel that it is somehow skewed towards Contrast, they have the power to change that.  Now, if someone tries to participate and Dave tells them "No thanks", then I agree we have a problem, but I don't hear anyone inferring that happened.                                              <br>                                              <br>                                             </div>Please, let's drop the conspiracy theories and focus on the tangible things that we can do to help an OWASP project to be more successful.  Help find more participants to drive diversity, update our brand usage guidelines to prevent abuse, enforce them widely, etc.  Thank you.                                             <br>                                             <br>                                            </div>~josh                                            <br>                                           </div>                                           <div>                                            <br>                                            <div>                                             On Thu, Nov 26, 2015 at 4:24 PM, Jim Manico                                              <span dir="ltr"><<a>jim.manico@owasp.org</a>></span> wrote:                                             <br>                                             <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                                              <div dir="auto">                                               <div>                                                Dinis,                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                Like a rare celestial moment when all the planets plus Pluto are aligned, I just read your email on the future of OWASP projects thinking, "Dinis is spot on".                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                Reflecting on projects I manage or work on...                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                The Java Encoder and HTML Sanitizer are likely best moved to Apache now that they have reached a measure of adoption and maturity. Apache would be a much better long term custodian. Perhaps the same for AppSensor, but not my project - just thinking out loud.                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                Other similar defensive projects are still being noodled on, so OWASP is a decent home for these research efforts.                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                The whole tools category is also something to consider. Dependency Check and of course ZAP are some of the best projects that OWASP offers, are they best served where they are today? Both have rich communities of developers but I don't see the foundation doing much to support these efforts.                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                ASVS has the opportunity to effect massive change, I would to love to see major investment and volunteer activity here. Pro tech writer, detailed discourses on each individual requirement, etc. If I was king (and I am not, at all) I would invest in ASVS on a 6 figure scale. (And who started ASVS? Jeff, Dave and Boberski, hat tip to such a marvelous idea). Or maybe moving ASVS to the W3C or IETF would help it grow?                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                The Proactive Controls was a pet project but as we approach 2.0 we have several active/awesome volunteers working on it. We will be making the doc "world editable" to make contributions easy. OWASP seems like a good home for such an awareness doc. Same with T10, especially if community edits are welcome.                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                Anyhow, I'm with you on this Dinis. Once a project starts to reach production quality, spinning off the project as an external project or moving it to a different foundation where managing production software or formal standards is their thing seems realistic.                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                I don't have all the answers here, but your email certainly resonated with me.                                               </div>                                               <div>                                                <br>                                               </div>                                               <div>                                                Aloha,                                               </div>                                               <div>                                                <div>                                                 --                                                </div>                                                <div>                                                 Jim Manico                                                </div>                                                <div>                                                 <div>                                                  <div style="word-wrap:break-word">                                                   <div>                                                    <span style="background-color:rgba(255,255,255,0)">Global Board Member</span>                                                   </div>                                                   <span style="background-color:rgba(255,255,255,0)">OWASP Foundation</span>                                                   <div>                                                    <a href="https://www.owasp.org/" style="background-color:rgba(255,255,255,0)" target="_blank"><font color="#000000">https://www.owasp.org</font></a>                                                   </div>                                                  </div>                                                 </div>                                                 <div>                                                  <span style="background-color:rgba(255,255,255,0)">Join me in Rome for AppSecEU 2016!</span>                                                 </div>                                                </div>                                               </div>                                               <div>                                                <br>On Nov 26, 2015, at 11:26 PM, Dinis Cruz <                                                <a>dinis.cruz@owasp.org</a>> wrote:                                                <br>                                                <br>                                               </div>                                               <blockquote>                                                <div>                                                 <div dir="ltr">                                                  Jim's reading of this situation is exactly my view on the value of the Contrast tool and how it has been 'pushing' the rules of engagement to an very 'fuzzy' moral/ethical/commercial limit :)                                                  <div>                                                   <br>                                                  </div>                                                  <div>                                                   As per my last email, a key problem here is the 'perceived expectation' of what is an OWASP project, and how it should be consumed.                                                  </div>                                                  <div>                                                   <br>                                                  </div>                                                  <div>                                                   If you look at the OWASP benchmark as a research project, then the only way it could be making the kind of claims it makes (and have credibility) is if it had evolved from OWASP, with its own (diverse) community                                                   </div>                                                 </div>                                                 <div>                                                  <br>                                                  <div>                                                   On 26 November 2015 at 21:01, Jim Manico                                                    <span dir="ltr"><<a>jim.manico@owasp.org</a>></span> wrote:                                                   <br>                                                   <blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">                                                    <div dir="auto">                                                     <div>                                                      I have a different take on this situation but my opinion is the "minority opinion". I will respect the rest of the boards take on this, but here is how I see it.                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      First of all, Jeff has stated that he feels I am attacking him personally from a past personal grudge, and frankly I do not fault him for that perspective since we definitely have history with conflict. So it's fair to take my opinion on this with a grain of salt.                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      I look at this situation from the perspective of a forensic investigator.                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      1) The Benchmark project had Contrast hooks and only Contrast hooks in it when I reviewed it so this leads me to believe that the project was clearly built with Contrast in mind from the ground up, at least in some way.                                                     </div>                                                     <div>                                                      3) Dave has a history of breaching his duty to be vendor neutral. He was gifted with a keynote in South Korea a few years ago, and used that opportunity to discuss and pitch Contrast, on stage, during a keynote - with Contrast specific slides. This is just supporting evidence of his intention at OWASP to push Contrast in ways that I think are against the intentions and goals of our foundation.                                                     </div>                                                     <div>                                                      3) Other experts have reviewed the project and felt that many of the tests were very slanted and almost contrived to support Contrast. I can drag those folks into this conversation, but I do not think that would help in any way. So it's fair to call this point heresy.                                                      </div>                                                     <div>                                                      4) I do not see this project as revolutionary, at all. Every vendor has their own test suite tuned for their tool. As the benchmark stands today, I see it as just another vendors product-specific benchmark. Mass collaboration from many vendors is not just a "nice to have" but a base requirement to get even close to useful for objective tool measurement.                                                     </div>                                                     <div>                                                      5) Jeff stating that his Marketing people went over the line is also an admission that - well, they went over the line. By the same token Jeff was in his booth at AppSec USA surrounded by benchmark marketing material, discussing this to prospects and he even asked me and Mr Coates to wade into this debate and support Dave. So to say he was not involved and it was only his marketing people seems a stretch at best.                                                     </div>                                                     <div>                                                      6) The Contrast marketing team was wandering around the conference zapping folks to get leads, and I asked them to stay in their booth, which is standard conference policy. These folks know better but are again going over the line to sell product at OWASP. There is a better way (like focusing on product capability and language support, have consistent + stellar customer service, have a humble and gracious attitude to all prospects and customers, actively participate in OWASP in a vendor neutral and community supportive way, etc).                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      Please note, I think Contrast is a decent tool, I've offered to resell in the past, and I have recommended it in certain situations - even after this situation arose. I'm stating this out of honestly and desire to put my cards on the table. I truly want Jeff and Dave to be successful. They have dedicated their lives to AppSec and if anyone should win big-time, I hope it's them. I even told Jeff I hope he hits the mother load and donates a little back to OWASP.                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      However, my instinct and evidence tell me that they both went over the line in the use of the OWASP brand to sell product.                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      Now, Jeff makes a good point. We as a board and staff are very poor at enforcing brand management policy and it's not fair to single out Contrast, when many other vendors violate the brand, IMO. Just google OWASP and watch the ads fly that use the OWASP name to sell product.                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      Also, any and every request that was made of Dave to adjust the project for the sake of vendor neutrality was taken very seriously. Regardless of Daves past intentions, he is clearly trying to do the right thing moving forward.                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      I look to "postels principle" in this situation (this is otherwise known as the "robustness principle" and dates back to the creation of TCP) . This is paraphrased as, "Be liberal in what you take from others but be conservative in what you dish out". So I think it's critical that OWASP and any OWASP resource present itself in a strict vendor neutral way. But unless OWASP wants to be much more "even" in the enforcement of brand policy across the board to all violators, we should be fairly lax in the enforcement of these issues from the outside world.                                                     </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      I am trying to be objective here. My trigonometry teacher once told me "I'd fail my mother" when I asked him if he would ever fail me (I was an A student). If my mother owned a security company and tried the same stunt, I'd have the same opinions about her actions as well.                                                      </div>                                                     <div>                                                      <br>                                                     </div>                                                     <div>                                                      So what next? Well hello from the other side. I'm going back to listening to Adele's new album where I can</div></div></blockquote></div></div></div></blockquote></div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div></div></div></div></blockquote></div></div></div></blockquote></div>