[Owasp-leaders] OWASP Benchmark project - potential conflict of interest
johanna curiel curiel
johanna.curiel at owasp.org
Thu Oct 1 02:47:31 UTC 2015
For some weird reason I have been kicked out to post on the mailing list,
on all it seems and I get some emails like the one Timo sent but I also
noticed I lost all track with the latest discussions.Especially when my
review has not been clearly mentioned here.
I would appreciate you replying to the list so people can read what is
I have asked Kate to check for me this issue
On Wed, Sep 30, 2015 at 9:08 PM, johanna curiel curiel <
johanna.curiel at owasp.org> wrote:
> Hi All
> From the beginning, in the last review I did for this project I mentioned
> this potential issue. The project by itself seems to try to accomplish
> something that could benefit researchers looking to test and 'benchmark'
> tools, but the approach and the accuracy of the test has been also my
> concern from day one. Attached you can find my review.
> What I don't like from this article is the sentence(:
> *It’s no surprise that Williams is promoting the OWASP benchmark as his
> company’s application testing tool *performs well given the parameters
> that OWASP created.
> Also the article mentions:
> "Benchmarks are important and I applaud the work OWASP has done. *Yet it
> is still early days for the benchmark, now in version 1.2 beta*. I expect
> it to become more accurate and the results presented more fairly in the
> future as other application security vendors and security experts dig in
> and help improve it.
> For some reason is my question: Is* the tool been promoted as if is quite
> 'ready' for benchmarking?* That is unfortunately not the case from the
> reviews we did. We can consider this at LAB stage, with a viable
> deliverable but not as a main and stable tool as you can read from the last
> review I did.
> One of the issues I found was:
> "In order to benchmark a tool, it is essential that the tool being tested
> (for example ZAP) can produce an output of its results in XML. If the
> tested DAST/SAST tool does not have a complete output of all its findings
> (like is the case of ZAP) OWASP Benchmark will be only able to ‘benchmark’
> these results, creating a limited view of the full capabilities of a
> SATS/DAST tool "
> I sustain my word that I really think the tool and it has potential to
> help researches benchmark their tools but it is clear that the tool is yet
> in a very early stage to do this reliable and it will require a lot of use
> and testing.
> On Wed, Sep 30, 2015 at 12:42 PM, Timo Goosen <timo.goosen at owasp.org>
>> I don't know enough about the matter to comment on this case, but I feel
>> that any situation where an OWASP project or any OWASP initiative for that
>> matter, is using OWASP to promote its own business interests should be
>> stopped. We need to get rid of bad apples in OWASP.
>> OWASP is becoming a brand if you would like to think of it that way and
>> we are going to see many more cases of people trying to use OWASP to spread
>> their business interests. At the end of the day everyone should be acting
>> with an attitude of:"Don't ask what OWASP can do for me, ask what I can do
>> for OWASP?"
>> On Wed, Sep 30, 2015 at 11:48 AM, psiinon <psiinon at gmail.com> wrote:
>>> So, a load of controversy about OWASP Benchmark on twitter, but no
>>> discussion on the leaders list :(
>>> Is this now the wrong place to discuss OWASP projects??
>>> On Thu, Sep 24, 2015 at 10:36 AM, psiinon <psiinon at gmail.com> wrote:
>>>> Hi folks,
>>>> I've got some concerns about the OWASP Benchmark project.
>>>> I _like_ benchmarks, and I'm very pleased to see an active OWASP
>>>> project focused on delivering one.
>>>> I think the project has some technical limitations, but thats fine
>>>> given the stage the project is at, ie _very_ early.
>>>> I dont think that any firm conclusions should be drawn from it until
>>>> its been significantly enhanced.
>>>> My concerns are around the marketing that one of the companies
>>>> sponsoring the Benchmark project has started using.
>>>> Here we have a company that leads an OWASP project that just happens to
>>>> show that their offering in this area appears to be _significantly_ better
>>>> than any of the competition.
>>>> Their recent press release stresses that its an OWASP project, make the
>>>> most of the fact that the US DHS helped fund it but make no mention of
>>>> their role in developing it.
>>>> Regardless of the accuracy of the results, it seems like a huge
>>>> conflict of interest :(
>>>> It appears that I'm not the only one with concerns related to the
>>>> What do other people think?
>>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>> OWASP ZAP <https://www.owasp.org/index.php/ZAP> Project leader
>>> OWASP-Leaders mailing list
>>> OWASP-Leaders at lists.owasp.org
>> OWASP-Leaders mailing list
>> OWASP-Leaders at lists.owasp.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OWASP-Leaders