[Owasp-leaders] OWASP Benchmark project - potential conflict of interest

Timo Goosen timo.goosen at owasp.org
Thu Oct 1 11:46:56 UTC 2015


Seems like Johanna can't post to the list. Below is her response that some
people missed.

Regards.
Timo

On Thu, Oct 1, 2015 at 3:08 AM, johanna curiel curiel <
johanna.curiel at owasp.org> wrote:

> Hi All
>
> From the beginning, in the last review I did for this project I mentioned
> this potential issue. The project by itself seems to try to accomplish
> something that could benefit researchers looking to test and 'benchmark'
> tools, but the approach and the accuracy of the test has been also my
> concern from day one. Attached you can find my review.
>
> What I don't like from this article is the sentence(:
> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet)
> *It’s no surprise that Williams is promoting the OWASP benchmark as his
> company’s application testing tool *performs well given the parameters
> that OWASP created.
>
> Also the article mentions:
> "Benchmarks are important and I applaud the work OWASP has done.  *Yet it
> is still early days for the benchmark, now in version 1.2 beta*. I expect
> it to become more accurate and the results presented more fairly in the
> future as other application security vendors and security experts dig in
> and help improve it.
>
> For some reason is my question: Is* the tool been promoted as if is quite
> 'ready' for benchmarking?* That is unfortunately not the case from the
> reviews we did. We can consider this at  LAB stage, with a viable
> deliverable but not as a main and stable tool as you can read from the last
> review I did.
>
> One of the issues I found was:
>
>
> "In order to benchmark a tool, it is essential that the tool being tested
> (for example ZAP) can produce an output of its results in XML. If the
> tested DAST/SAST tool does not have a complete output of all its findings
> (like is the case of ZAP) OWASP Benchmark will be only able to ‘benchmark’
> these results, creating a limited view of the full capabilities of a
> SATS/DAST tool "
>
>
> I sustain my word that I really think the tool and it has potential to
> help researches benchmark their tools but it is clear that the tool is yet
> in a very early stage to do this reliable and it will require a lot of use
> and testing.
>
>
> regards
>
>
> Johanna
>
>
>
> On Wed, Sep 30, 2015 at 12:42 PM, Timo Goosen <timo.goosen at owasp.org>
> wrote:
>
>> I don't know enough about the matter to comment on this case, but I feel
>> that any situation where an OWASP project or any OWASP initiative for that
>> matter, is using OWASP to promote its own business interests should be
>> stopped.  We need to get rid of bad apples in OWASP.
>>
>> OWASP is becoming a brand if you would like to think of it that way and
>> we are going to see many more cases of people trying to use OWASP to spread
>> their business interests. At the end of the day everyone should be acting
>> with an attitude of:"Don't ask what OWASP can do for me, ask what I can do
>> for OWASP?"
>>
>>
>>
>> Regards.
>> Timo
>>
>> On Wed, Sep 30, 2015 at 11:48 AM, psiinon <psiinon at gmail.com> wrote:
>>
>>> So, a load of controversy about OWASP Benchmark on twitter, but no
>>> discussion on the leaders list :(
>>> Is this now the wrong place to discuss OWASP projects??
>>>
>>> Simon
>>>
>>>
>>> On Thu, Sep 24, 2015 at 10:36 AM, psiinon <psiinon at gmail.com> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> I've got some concerns about the OWASP Benchmark project.
>>>>
>>>> I _like_ benchmarks, and I'm very pleased to see an active OWASP
>>>> project focused on delivering one.
>>>> I think the project has some technical limitations, but thats fine
>>>> given the stage the project is at, ie _very_ early.
>>>> I dont think that any firm conclusions should be drawn from it until
>>>> its been significantly enhanced.
>>>>
>>>> My concerns are around the marketing that one of the companies
>>>> sponsoring the Benchmark project has started using.
>>>>
>>>> Here we have a company that leads an OWASP project that just happens to
>>>> show that their offering in this area appears to be _significantly_ better
>>>> than any of the competition.
>>>> Their recent press release stresses that its an OWASP project, make the
>>>> most of the fact that the US DHS helped fund it but make no mention of
>>>> their role in developing it.
>>>>
>>>> Regardless of the accuracy of the results, it seems like a huge
>>>> conflict of interest :(
>>>>
>>>> It appears that I'm not the only one with concerns related to the
>>>> project:
>>>>
>>>> https://www.veracode.com/blog/2015/09/no-one-technology-silver-bullet
>>>>
>>>> What do other people think?
>>>>
>>>> Cheers,
>>>>
>>>> Simon
>>>>
>>>> --
>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>>
>>>
>>>
>>>
>>> --
>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>>
>>> _______________________________________________
>>> OWASP-Leaders mailing list
>>> OWASP-Leaders at lists.owasp.org
>>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>>
>>>
>>
>> _______________________________________________
>> OWASP-Leaders mailing list
>> OWASP-Leaders at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-leaders
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-leaders/attachments/20151001/1863229e/attachment-0001.html>


More information about the OWASP-Leaders mailing list