[Owasp-leaders] FW: REQUESTFOR DECISION/CALL FOR CONTRIBUTIONS TO UPDATE THE ASSESSMENTCRITERIA
mtesauro at gmail.com
Sun Mar 8 23:55:54 EDT 2009
You raise a situation I'd not considered - primarily because the
situation you describe is not what the "downgrade to Alpha" idea was
created to handle. At least that's not my understanding.
The discussion of downgrading to Alpha grew out of the committee trying
to figure out how to handle non-responsive project leads and/or dead
projects which had never been assessed. In the case that:
* a project lead does not respond to queries
* the mail list has prolonged inactivity
* the project pre-dates the assessment criteria
* and a project appears to be otherwise be dead or no longer relevant
Then moving them to Alpha seemed like an appropriate response. Alpha is
a very open space for projects and good for holding projects of unknown
status in the near term.
Considering the idea of appending a version to the ratings (props to
Mike), I can't see a reason to _ever_ downgrade a project in your
situation. Obviously you'll respond to queries so I can't image your
project being called dead. And as you stated below, if a project lead
sees value in going from beta criteria v1.0 to beta criteria 2.0, they
are free to make that choice and do the work.
To me, the idea of was to keep the project page from suffering from the
"sourceforge" effect where projects get created, possibly releases
some early code and then go stagnant. Going forward, I suspect that
we'll have to figure out how to determine that a project should be
archived and/or deprecated. I'm not in favor of removing projects -
just making the in-active, no longer relevant ones less prominent. My
example has been a PHP 3 secure coding guide. Might be a decent basis
for an update to PHP 5 but I pity the person coding in PHP 3.
As to your and Buanzo's projects (and likely others) not neatly fitting
into the Tool or Document bucket, its obvious from the feedback we've
gotten that this situation will need to be considered. Pravir has
posted a sound idea for this and I suspect this will be discussed in
tomorrow's committee meeting.
Much like Buanzo's, your project may not easily be categorized but that
doesn't diminish its value. It simply means that we need to better
figure out how to handle those that are not clearly tools or documents
- which is fantastic because that is why we asked for feedback.
-- Matt Tesauro
OWASP Live CD Project Lead
http://mtesauro.com/livecd/ - Documentation Wiki
Stephen Craig Evans wrote:
>>From Paulo's original email:
> "I would like to inform you that the Project’s Committee propose that,
> as soon as we finish this discussion, we establish as a rule to apply
> to all OWASP Projects that the quality categorization must respect the
> revised assessment criteria which eventually will mean that all
> projects not assessed under these rules will be placed under Alpha
> Quality status."
> I do not agree with this and I pretty much agree with what Mike has
> suggested with the following modification:
> The existing Assessment Criteria is version 1.0. The new one under
> current discussion could be version 2.0.
> As Mike suggested, the main project page should prominently display
> Quality Level with Assessment Criteria version number (and hotlinks to
> the definitions of those 2).
> For example, my SoC 2008 project, Securing WebGoat using ModSecurity,
> automatically gets the designation Tools Project with rating Beta
> Quality, Assessment Criteria version 1.0.
> If it does not meet the new standards and I want to meet them, then
> I'll do the necessary work and then it will be given a rating of Beta
> Quality, Assessment Criteria version 2.0.
> Doing it this way:
> - We avoid all of the hassle of going back and re-assessing projects
> previous to version 2.0
> - We don't lessen the value of version 1.0 projects - an insult the
> project owneres - by degrading them to Alpha Quality
> - We give project owners the option and opportunity to upgrade their
> version 1.0 projects to version 2.0
> - We have a flexible process that allows for future Assessment Criteria versions
> Hey Roomie!
> To address your and Matt's comments:
>> With all this, I basicly want to say that mine (and other projects, too) are not necessarily 'tools'
>> per se, and maybe there is need to assess this situation more throughly. I call for help in this
>> matter, or for someone to hit me in the face with a 'hey, you got it wrong, Buanzo! this is the
>> deal' explanation :P
> This time no smacking is needed. I would agree that your project
> isn't really a tool nor is it a document. Its a specification with a
> reference implementation. How that fits the assessment criteria, I
> honestly don't know but it is something we will have to account for
> going forward.
> Yes, your project is in a gray area betwen Tools and Documentation. I
> consider my project mainly as a research project since it evolved into
> using Lua scripts in ModSecurity, which really hasn't been done
> before. I provide ModSecurity rulesets and complete fully functional
> Lua scripts, plus I rebuilt the Lua standalone executable and shared
> object libraries from source to include extra functionality. It's not
> a Tools project but not a Documentation project, but it's fairly clear
> that it is not a Tools project so it falls under Documentation.
> If your project is a reference implementation, then IMHO it falls more
> under Documentation category. Yes, "Documentation", meaning non-Tools,
> might not be the best name and that could be changed; however, I can't
> think of a name better than "Non-Tools" off the top of my head.
> So, under my proposed modification at the beginning of this email,
> your project could either be Beta Tools Quality, Assessment Criteria
> version 1.0 -or- Beta Documentation Quality, Assessment Criteria
> version 2.0.
> Your Season of Quality suggestions still hold; in my case, I could do
> the same and offer a VMWare image with Ubuntu, WebGoat, ModSecurity,
> and include my re-built Lua version (with MD5 and SQL database access
> as extra features), my Lua scripts, and ModSec rulesets.
> Again, your project is in a gray area. I don't think there should be
> an extra category besides Tools and Documentation (even though
> "Documentation" could be renamed) - it will just introduce more room
> for confusion. But, you can always choose what category you want for
> your project based on the criteria.
> On Sun, Mar 8, 2009 at 12:25 PM, Matt Tesauro <mtesauro at gmail.com> wrote:
>> Mike Boberski wrote:
>>> The only way to eat an elephant is one bite at a time.
>>> Let us start with the first bite.
>>> Re: "Please note that not all the projects below have been evaluated
>>> under this criteria and might be re-classified once that process is
>>> completed" which is now on each tab of the projects page. We need to
>>> discuss this!
>>> Overturning verdicts/assessments does not provide consumers in any
>>> context with confidence that the rating organization has its stuff
>>> together or that its ratings mean anything. It is also completely
>>> demoralizing to contributors.
>>> I wrote earlier:
>>> > should simply put dates against ratings, and identify the criteria
>>> > version that a project was assessed against, then leave that rating
>>> > alone as the criteria continues to evolve over time. That is what
>>> > more well-established and formal testing programs for instance like
>>> > Common Criteria and FIPS 140 do.
>>> I propose the following actions:
>>> 1. Delete "Please note that not all the projects below have been
>>> evaluated under this criteria and might be re-classified once that
>>> process is completed" from the tabs on the projects page.
>> I agree with this. I'd rather have to clear up and answer specific
>> questions then globally spread doubt. Hell, its a wiki so why not make
>> the change, its all versioned anyway..
>>> 2. Append to each project's short description on the projects page
>>> "(Assessment Criteria <version>)". Going with version only instead of
>>> version and date will simplify matters.
>> I also like this as it both shows per project where they are and also
>> (by its lack) would show projects that pre-date the criteria. This is
>> useful information to both internal and external audiences.
>>> 3. (Optional step) On each project's project page, wherever it currently
>>> identifies its release status at the top of the page, append
>>> "(Assessment Criteria <version>)"
>> I also think this is a stellar idea.
>>> I am assuming you're maintaining version control for project assessment
>>> criteria. If not the time is now. Mark the current criteria v1.0, make
>>> sure it's saved off and made accessible on the site somewhere, and
>>> assign versions when new versions are released as time goes on.
>> I haven't heard an explicit version number used - most frequently it is
>> spoken about with the SoC as its context.
>> Since the one used for 2008 SoC is the first assessment criteria, v1.0
>> makes sense to me.
>>> If there has been no criteria versioning to date, or if you can tell me
>>> the current version, I will go an update all listings on the projects
>>> page, then you guys/whomever if you want to perform optional step (3)
>>> above, can email project leads and ask them to update their project page
>>> to match.
>> Again, v1.0 makes the most sense for the criteria version number. Paulo
>> will hopefully chime in if I'm incorrect in my thinking.
>>> If there's agreement, let me know what criteria version to put, and I'll
>>> take care of updating the projects page listings. Then we can move onto
>>> the next bite.
>> If you don't get an explicit request to NOT do this by the time you wake
>> up on Tuesday, March 10th (whatever the timezone), then consider
>> agreement to be reached. I've CC'ed the Global Project Committee in
>> case they disagree with this.
>> -- Matt Tesauro
>> OWASP Live CD Project Lead
>> http://mtesauro.com/livecd/ - Documentation Wiki
>>> OWASP-Leaders mailing list
>>> OWASP-Leaders at lists.owasp.org
>> OWASP-Leaders mailing list
>> OWASP-Leaders at lists.owasp.org
More information about the OWASP-Leaders