[Owasp-leaders] FW: REQUESTFOR DECISION/CALL FOR CONTRIBUTIONS TO UPDATE THE ASSESSMENTCRITERIA
Stephen Craig Evans
stephencraig.evans at gmail.com
Sun Mar 8 01:41:29 EST 2009
>From Paulo's original email:
"I would like to inform you that the Project’s Committee propose that,
as soon as we finish this discussion, we establish as a rule to apply
to all OWASP Projects that the quality categorization must respect the
revised assessment criteria which eventually will mean that all
projects not assessed under these rules will be placed under Alpha
I do not agree with this and I pretty much agree with what Mike has
suggested with the following modification:
The existing Assessment Criteria is version 1.0. The new one under
current discussion could be version 2.0.
As Mike suggested, the main project page should prominently display
Quality Level with Assessment Criteria version number (and hotlinks to
the definitions of those 2).
For example, my SoC 2008 project, Securing WebGoat using ModSecurity,
automatically gets the designation Tools Project with rating Beta
Quality, Assessment Criteria version 1.0.
If it does not meet the new standards and I want to meet them, then
I'll do the necessary work and then it will be given a rating of Beta
Quality, Assessment Criteria version 2.0.
Doing it this way:
- We avoid all of the hassle of going back and re-assessing projects
previous to version 2.0
- We don't lessen the value of version 1.0 projects - an insult the
project owneres - by degrading them to Alpha Quality
- We give project owners the option and opportunity to upgrade their
version 1.0 projects to version 2.0
- We have a flexible process that allows for future Assessment Criteria versions
To address your and Matt's comments:
> With all this, I basicly want to say that mine (and other projects, too) are not necessarily 'tools'
> per se, and maybe there is need to assess this situation more throughly. I call for help in this
> matter, or for someone to hit me in the face with a 'hey, you got it wrong, Buanzo! this is the
> deal' explanation :P
This time no smacking is needed. I would agree that your project
isn't really a tool nor is it a document. Its a specification with a
reference implementation. How that fits the assessment criteria, I
honestly don't know but it is something we will have to account for
Yes, your project is in a gray area betwen Tools and Documentation. I
consider my project mainly as a research project since it evolved into
using Lua scripts in ModSecurity, which really hasn't been done
before. I provide ModSecurity rulesets and complete fully functional
Lua scripts, plus I rebuilt the Lua standalone executable and shared
object libraries from source to include extra functionality. It's not
a Tools project but not a Documentation project, but it's fairly clear
that it is not a Tools project so it falls under Documentation.
If your project is a reference implementation, then IMHO it falls more
under Documentation category. Yes, "Documentation", meaning non-Tools,
might not be the best name and that could be changed; however, I can't
think of a name better than "Non-Tools" off the top of my head.
So, under my proposed modification at the beginning of this email,
your project could either be Beta Tools Quality, Assessment Criteria
version 1.0 -or- Beta Documentation Quality, Assessment Criteria
Your Season of Quality suggestions still hold; in my case, I could do
the same and offer a VMWare image with Ubuntu, WebGoat, ModSecurity,
and include my re-built Lua version (with MD5 and SQL database access
as extra features), my Lua scripts, and ModSec rulesets.
Again, your project is in a gray area. I don't think there should be
an extra category besides Tools and Documentation (even though
"Documentation" could be renamed) - it will just introduce more room
for confusion. But, you can always choose what category you want for
your project based on the criteria.
On Sun, Mar 8, 2009 at 12:25 PM, Matt Tesauro <mtesauro at gmail.com> wrote:
> Mike Boberski wrote:
>> The only way to eat an elephant is one bite at a time.
>> Let us start with the first bite.
>> Re: "Please note that not all the projects below have been evaluated
>> under this criteria and might be re-classified once that process is
>> completed" which is now on each tab of the projects page. We need to
>> discuss this!
>> Overturning verdicts/assessments does not provide consumers in any
>> context with confidence that the rating organization has its stuff
>> together or that its ratings mean anything. It is also completely
>> demoralizing to contributors.
>> I wrote earlier:
>> > should simply put dates against ratings, and identify the criteria
>> > version that a project was assessed against, then leave that rating
>> > alone as the criteria continues to evolve over time. That is what
>> > more well-established and formal testing programs for instance like
>> > Common Criteria and FIPS 140 do.
>> I propose the following actions:
>> 1. Delete "Please note that not all the projects below have been
>> evaluated under this criteria and might be re-classified once that
>> process is completed" from the tabs on the projects page.
> I agree with this. I'd rather have to clear up and answer specific
> questions then globally spread doubt. Hell, its a wiki so why not make
> the change, its all versioned anyway..
>> 2. Append to each project's short description on the projects page
>> "(Assessment Criteria <version>)". Going with version only instead of
>> version and date will simplify matters.
> I also like this as it both shows per project where they are and also
> (by its lack) would show projects that pre-date the criteria. This is
> useful information to both internal and external audiences.
>> 3. (Optional step) On each project's project page, wherever it currently
>> identifies its release status at the top of the page, append
>> "(Assessment Criteria <version>)"
> I also think this is a stellar idea.
>> I am assuming you're maintaining version control for project assessment
>> criteria. If not the time is now. Mark the current criteria v1.0, make
>> sure it's saved off and made accessible on the site somewhere, and
>> assign versions when new versions are released as time goes on.
> I haven't heard an explicit version number used - most frequently it is
> spoken about with the SoC as its context.
> Since the one used for 2008 SoC is the first assessment criteria, v1.0
> makes sense to me.
>> If there has been no criteria versioning to date, or if you can tell me
>> the current version, I will go an update all listings on the projects
>> page, then you guys/whomever if you want to perform optional step (3)
>> above, can email project leads and ask them to update their project page
>> to match.
> Again, v1.0 makes the most sense for the criteria version number. Paulo
> will hopefully chime in if I'm incorrect in my thinking.
>> If there's agreement, let me know what criteria version to put, and I'll
>> take care of updating the projects page listings. Then we can move onto
>> the next bite.
> If you don't get an explicit request to NOT do this by the time you wake
> up on Tuesday, March 10th (whatever the timezone), then consider
> agreement to be reached. I've CC'ed the Global Project Committee in
> case they disagree with this.
> -- Matt Tesauro
> OWASP Live CD Project Lead
> http://mtesauro.com/livecd/ - Documentation Wiki
>> OWASP-Leaders mailing list
>> OWASP-Leaders at lists.owasp.org
> OWASP-Leaders mailing list
> OWASP-Leaders at lists.owasp.org
More information about the OWASP-Leaders