[Owasp-board] On the idea of Alpha, Beta and Release quality for Projects

Jim Manico jim.manico at owasp.org
Sun May 4 22:19:50 UTC 2014


> Well , how do you make sure you have an "objective" technical review?

By measuring objective things like time to fix bugs, number of unfixed
bugs, status of documentation, time since last substantial update, number
of updates, etc for starters. These are things we can review with minimal
tech expertise.

Admittedly, it's takes a lot of technical expertise to evaluate other
quality characteristics of OWASP projects. One if my beefs with ESAPI is
that they use the same instance of SecureRandom for all random number
generation, which is deterministic. This is really bad! Security libraries
need good random number generation or the entire crypto API is undermined!
Admittedly, only a Java and Security expert would even recognize these
problems. That's not an easy problem to deal with, but maybe we can file
those under "number of bugs, etc". We also have live CVE's of broken crypto
in ESAPI. These are serious unfixed problems....

••• My goal here is to be honest about OWASP project quality •••

If we can't measure project quality fairly, I'd much rather just say that
instead of making claims of flagship (and production quality).

Several large companies have done reviews of our flagships and deemed them
far from production quality when we claimed they were. This ties into
fundraising. Fundraising is one of my roles at board members.

So I will support any honest direction you want to go in Johanna. Does my
position here make more sense?

With respect,
--
Jim Manico
@Manicode
(808) 652-3805

On May 4, 2014, at 4:54 PM, johanna curiel curiel <johanna.curiel at owasp.org>
wrote:

I think objective technical review is much more important than a "community
vote" where the system can easily be gamed.

Well , how do you make sure you have an "objective" technical review?



On Thu, Apr 17, 2014 at 12:00 PM, Jim Manico <jim.manico at owasp.org> wrote:

>  I see the community vote for project review to be a "popularity vote" and
> not a "quality review". I would recommend having three main technical
> reviewers: One for assessment projects, one for code projects and another
> for documentation. These three need very different criteria for quality and
> flagship type classification.
>
> I think objective technical review is much more important than a
> "community vote" where the system can easily be gamed.
>
>
> > Testing and fixing is the only way to verify how stable a release is.
>
> An easier way is to review google code or GIT and check out the history of
> reported and fixed bugs for tool and code projects.
>
> For example take ESAPI for Java: They claim it's production but its full
> of existing critical bugs:
> https://code.google.com/p/owasp-esapi-java/issues/list?can=2&q=&sort=priority&colspec=ID%20Type%20Status%20Priority%20Milestone%20Component%20Owner%20Summary
>
> Take OWASP Java Encoder where historically we have 7 bugs reported..
> https://code.google.com/p/owasp-java-encoder/issues/list?can=1&q=&colspec=ID+Type+Status+Priority+Milestone+Owner+Summary&cells=tiles
>
> But worked hard to fix them all:
> https://code.google.com/p/owasp-java-encoder/issues/list
>
> These are much more valuable data points than community vote, in my
> opinion.
>
> But I do think that USERS of the project voting is valuable.
>
> My 1 cent,
> Jim
>
>
> On 4/16/14, 7:34 PM, johanna curiel curiel wrote:
>
> Well, that's why is so important to have a review system and community
> providing input
> How can we know a project release is what they claim?
>
>  Just as Microsoft can claims "beta" or final release still,  many
> patches have to been release afterwards to fix issues.
>
>  If a project claims is "beta" or final  release but is full with bugs it
> will find out by itself.
>  Testing and fixing is the only way to verify how stable a release is.
>
>  A new project with a first time release most of the time will be between
> alpha and beta.
> After a clear use of the project and feedback from the community, and
> multiple fixes, it can move to a stable release
>
>  regards
>
>  Johanna
>
>
> On Wed, Apr 16, 2014 at 9:15 PM, Jim Manico <jim.manico at owasp.org> wrote:
>
>>  What happens when products classify themselves as release quality when
>> they are really alpha quality?
>> - Jim
>>
>>
>>
>> On 4/14/14, 6:48 PM, Samantha Groves wrote:
>>
>> Yes, please just to clarify, we already have this. Project products must
>> self-classify as release, beta or alpha quality.
>>
>>
>> On Mon, Apr 14, 2014 at 3:50 PM, Dinis Cruz <dinis.cruz at owasp.org> wrote:
>>
>>> Hi Jim, when you in
>>> http://lists.owasp.org/pipermail/owasp-board/2014-April/013567.htmlsay:
>>>
>>>  *Something like: *
>>>>
>>>>    - *OWASP Project - Release Quality *
>>>>
>>>>
>>>>    - *OWASP Product - BETA Quality *
>>>>
>>>>
>>>>    - *OWASP Product - ALPHA Quality *
>>>>
>>>>
>>>>    - *Incubator Project*
>>>>
>>>>
>>>  You mean something like:
>>> https://www.owasp.org/index.php/Assessing_Project_Releases
>>>
>>>  And when talking about OWASP Project Health, what about:
>>> https://www.owasp.org/index.php/Assessing_Project_Health
>>>
>>>  Note that those are pages created by Paulo in 2009, Johanna and
>>> Samantha used them has bases of their more recent project assessment work
>>>
>>>  Dinis
>>>
>>
>>
>>
>>  --
>>
>> *Samantha Groves, MBA*
>>
>> *OWASP Projects Manager*
>>
>>
>>  The OWASP Foundation
>>
>> Phoenix, USA
>>
>> Email: samantha.groves at owasp.org
>>
>> Skype: samanthahz
>>
>>
>>  OWASP Global Projects<https://www.owasp.org/index.php/Category:OWASP_Project>
>>
>> Book a Meeting with Me <http://goo.gl/mZXdZ>
>>
>> OWASP Contact US Form <http://owasp4.owasp.org/contactus.html>
>>
>> New Project Application Form <http://www.tfaforms.com/263506>
>>
>>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20140504/87d5d9f8/attachment.html>


More information about the Owasp-board mailing list