[Owasp-testing] OWASP Testing Guide v3: introduce section for Software System Upgrade and Migration testing

Thu May 22 11:33:47 EDT 2008

As Dan and I have said, as past leaders of the guide, the guide is focused
on Black box, Grey box pen testing. It was never intended to cover testing
during the SDLC. Branching out too much shall dilute and weaken the core aim
of the book. Secure SDLC is an entirely different subject.

A great idea would be to document SDLC testing phases, where they should be
done, how to do them, requirements definition of a testing plan, integration
of security testing into functional testing during the SDLC and the
assignment and remediation of issues/bugs found during this task. This would
be a great addition to the Development Guide, Testing Guide, Code Review
Guide.....a "Secure SDLC Leading Practice Methodology" Book but IMHO adding
this to the Testing guide is inappropriate.

On 22/05/2008, Basim Baassiri <basim at baassiri.ca> wrote:
>
> Thank you all for the discussion,
>
> I wanted to bring an attention to this topic as it typically it is an after
> thought in most organizations. Like I mentioned earlier, with my experience,
> I've seen functional testing passing on a fresh install but once an app was
> taken from the previous deployed state to the new state it would completely
> fail.  In addition, code changes made to newer versions that checks a state
> expects to be one format and the data is another format and as a result it
> displays sensitive information. It can be as small as a typo
>
> I guess the above examples demonstrates more poor programming techniques
> but as quality assurance testers/professionals it our job to cover all of
> the angles.
>
> I Look forward to participating with you all in this guide.
>
> Basim
>
> On Thu, May 22, 2008 at 10:01 AM, Daniel Cuthbert <
> daniel.cuthbert at owasp.org> wrote:
>
>> Problem is going off scope as to what the Testing guide covers. I agree
>> with you that we need to ensure that all risks have to be covered, the
>> Testing guide was always a method of looking at an applications security
>> from a Black/White hat perspective. There are a load of guides out there
>> that explain how to look at the infrastructure side of testing, and whilst
>> we can reference them, sticking to the application layer is more advisable,
>> knowing how hard this guide is to manage and control.
>>
>> It's an interesting area though
>>
>>
>>
>>  On 22 May 2008, at 3:06 PM, Dave van Stein wrote:
>>
>> Eoin
>>
>>> I sense scope creep here :)
>>> Patch management in an application security testing guide?
>>>
>>
>> Not at all actually. The discussion is where application security stops
>> and what to do with known vulnerability risks beyond it. I think most of us
>> agree that these things do not belong in a application security
>> manual/framework, but I think it's also not a good idea to just completely
>> ignore them.
>> Like I said, the least we can do is make a (small) note (with reference)
>> when vulnerabilities are identified which are beyond the scope
>> of application security.
>> Ignorance should be avoided at all times.
>>
>>
>>> Also upgrade and migration testing, What are we actually going to test
>>> for here? If there is an application upgrade one does not simply test the
>>> upgraded piece [..]
>>>
>>  Migration testing would cover the same suite of tests as a regular
>>> application security test, but there may be some security review if the task
>>> was more than blackbox. SO we would look at config files etc. but testing
>>> specific to migration as opposed to testing to see if something is secure,
>>> the difference is minimal.
>>>
>> I agree. As a functional tester I have had these situations before and the
>> sensible way to scope your testplan is to do a risk analysis and decide
>> whether a regression test is enough or a complete functional test is
>> necessary. The same goes for security testing in my opinion.
>>
>> Dave
>>
>>
>>
>>>  On 21/05/2008, Dave van Stein <dvstein at gmail.com> wrote:
>>>>
>>>> Basim,
>>>>
>>>> I agree. In the mentioned case of upgrade en migration testing we could
>>>> add a note that at least patch management, authentication & access control
>>>> and configuration files should get extra attention to limit possible
>>>> vulnerabilities.
>>>>
>>>> regards, dave
>>>>
>>>>
>>>> 2008/5/21 Basim Baassiri <basim at baassiri.ca>:
>>>>
>>>>> I agree with your points Dave, there should be mention of this topic
>>>>> (upgrade and migration testing) when the scope of the manual has been
>>>>> reached.  Something along lines of "Although this topic is not in the scope
>>>>> of the manual, we feel that it should be mentioned so it is not an oversight
>>>>> when doing testing"
>>>>>
>>>>> Thanks again for all of your feedback
>>>>>
>>>>> Looking forward to helping out with the new guide
>>>>>
>>>>> Basim
>>>>>
>>>>>  On Tue, May 20, 2008 at 3:40 PM, Dave van Stein <dvstein at gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>>> All,
>>>>>>
>>>>>> I think that setting the scope and boundaries is one of the hardest
>>>>>> things concerning creating an application security testing manual. In the
>>>>>> case described above/below it handles the boundary between testing and SDLC,
>>>>>> but the same problem arises when defining the boundary between for example
>>>>>> application testing and ('classic') penetration testing. Clearly the goal of
>>>>>> OWASP is not testing things like the existence of open SSH sockets, but
>>>>>> things like command injection can be used to gain root access on a
>>>>>> webserver; where does application testing stop in that case ?
>>>>>>
>>>>>> On the other hand is limiting the scope to far not good either. In my
>>>>>> opinion the testing manual should at least give some information about risks
>>>>>> and things to improve when a 'boundary' is crossed.
>>>>>>
>>>>>> regards, Dave
>>>>>>
>>>>>> 2008/5/20 kevin horvath <kevin.horvath at gmail.com>:
>>>>>>
>>>>>> I agree that this is more on the SDLC side then the actual security
>>>>>>> testing of an application.  Secure SDLC testing would be a nice
>>>>>>> project to start for OWASP but I think it would be too much to
>>>>>>> incorporate into this project.
>>>>>>>
>>>>>>> On Tue, May 20, 2008 at 10:42 AM, Eoin <eoin.keary at owasp.org> wrote:
>>>>>>> > To Me this sounds more like Secure SDLC Testing and not pen
>>>>>>> testing, not
>>>>>>> > that the guide is strictly pen testing.
>>>>>>> >
>>>>>>> > Secure SDLC testing, test plans relating to functional requirements
>>>>>>> > etc could be a project on its own? Maintenance lifecycle,
>>>>>>> regression and
>>>>>>> > root cause analysis relating to the software development lifecycle
>>>>>>> can (and
>>>>>>> > should) involve an element of security. This area is very large and
>>>>>>> needs to
>>>>>>> > be comprehensively discussed and is a valid candidate for its own
>>>>>>> guide?
>>>>>>> >
>>>>>>> > I would be concerned that the guide may be stretching itself and
>>>>>>> covering
>>>>>>> > too many things rather than covering fewer things in a more
>>>>>>> complete manner.
>>>>>>> >
>>>>>>> > Eoin
>>>>>>> >
>>>>>>> > OWASP Code Review Guide Lead
>>>>>>> >
>>>>>>> >
>>>>>>> > On 20/05/2008, Basim Baassiri <basim at baassiri.ca> wrote:
>>>>>>> >>
>>>>>>> >> Has anyone had the chance to read below
>>>>>>> >>
>>>>>>> >> Basim
>>>>>>> >>
>>>>>>> >> On Mon, May 12, 2008 at 12:09 AM, Basim Baassiri <
>>>>>>> basim at baassiri.ca>
>>>>>>> >> wrote:
>>>>>>> >>>
>>>>>>> >>> Hello Matteo and all,
>>>>>>> >>>
>>>>>>> >>> When you say new concept, I'm assuming a new concept for the
>>>>>>> guide. As to
>>>>>>> >>> where it fits as part of the index, I can't clearly see it at
>>>>>>> this time.  To
>>>>>>> >>> be clearer on the term Software System upgrades and migrations,
>>>>>>> please see
>>>>>>> >>> my definition below.
>>>>>>> >>>
>>>>>>> >>> Software System Upgrade:
>>>>>>> >>> Basically, is testing the end result of the the data and
>>>>>>> application when
>>>>>>> >>> a patch/upgrade/hotfix has been applied to it (Upgrade path
>>>>>>> Testing)
>>>>>>> >>>
>>>>>>> >>> Typically, testing this process is as follows
>>>>>>> >>> 1. installation verification
>>>>>>> >>> Did the installation of the patches/upgrade executable/script to
>>>>>>> the
>>>>>>> >>> existing system cause an failures. In the context of a web
>>>>>>> application is
>>>>>>> >>> there exceptions/failures in the application log
>>>>>>> >>>
>>>>>>> >>> 2. Database verification (if required, but typically required in
>>>>>>> majority
>>>>>>> >>> of web applications)
>>>>>>> >>> This is a two step process where a tester would look to see
>>>>>>> (either the
>>>>>>> >>> output of the upgrade script or database logs) if any error
>>>>>>> occurred in the
>>>>>>> >>> logs
>>>>>>> >>> Second, a capture of a subset of a data from before and after
>>>>>>> would be
>>>>>>> >>> done to analyze transformations of data
>>>>>>> >>>
>>>>>>> >>> 3. Does the application behave the same way it did before
>>>>>>> (Functional
>>>>>>> >>> testing)
>>>>>>> >>>
>>>>>>> >>> As for migrations, this is defined by completely moving over the
>>>>>>> another
>>>>>>> >>> system.
>>>>>>> >>> Examples of migrations:
>>>>>>> >>> application A uses MySQL Database -> Apply migration ->
>>>>>>> application A now
>>>>>>> >>> uses PostGres database
>>>>>>> >>> application B uses PHP -> Apply migration -> application B now
>>>>>>> uses Java
>>>>>>> >>>
>>>>>>> >>> The reason I mention the above examples is because I have been
>>>>>>> exposed to
>>>>>>> >>> this scenario where it was expected that the application
>>>>>>> continues to behave
>>>>>>> >>> to same after the transformation
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> Thats the background of upgrade and migration testings
>>>>>>> >>>
>>>>>>> >>> I guess the next question is how does the above apply to security
>>>>>>> testing
>>>>>>> >>>
>>>>>>> >>> I've seen changes to table schemas where by the code is not
>>>>>>> reflective of
>>>>>>> >>> the change in the database and as a result "private" or sensitive
>>>>>>> data has
>>>>>>> >>> been exposed.
>>>>>>> >>>
>>>>>>> >>> Not sure where this falls under the current index.  This topic is
>>>>>>> not
>>>>>>> >>> commonly thought of in both the development and testing teams.
>>>>>>> >>>
>>>>>>> >>> Looking for feedback from anyone
>>>>>>> >>>
>>>>>>> >>> Cheers
>>>>>>> >>>
>>>>>>> >>> Basim
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> On Thu, May 8, 2008 at 12:00 PM, Basim Baassiri <
>>>>>>> basim at baassiri.ca>
>>>>>>> >>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> Hello all
>>>>>>> >>>>
>>>>>>> >>>> I've been watching the list for about 6 months now
>>>>>>> >>>>
>>>>>>> >>>> Just wanted to introduce myself
>>>>>>> >>>>
>>>>>>> >>>> My Background
>>>>>>> >>>> I am Computer Science graduate since 2000 and been doing
>>>>>>> software
>>>>>>> >>>> testing since 2004.  I'm currently in a QA lead role in a
>>>>>>> startup,
>>>>>>> >>>> www.overlay.tv.  I recently completed certification from
>>>>>>> cstb.ca.  I've done
>>>>>>> >>>> upgrade and migration testing as well as web application testing
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>> I'm noticing that the heading for the new v3 guide contain a
>>>>>>> wide range
>>>>>>> >>>> of testing activities and areas.  I would like to propose adding
>>>>>>> the
>>>>>>> >>>> following section
>>>>>>> >>>>
>>>>>>> >>>> Software System Upgrade and Migration
>>>>>>> >>>>
>>>>>>> >>>> Looking forward to your thoughts
>>>>>>> >>>>
>>>>>>> >>>> Basim
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>
>>>>>>> >> _______________________________________________
>>>>>>> >> Owasp-testing mailing list
>>>>>>> >> Owasp-testing at lists.owasp.org
>>>>>>> >> https://lists.owasp.org/mailman/listinfo/owasp-testing
>>>>>>> >>
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > Eoin Keary OWASP - Ireland
>>>>>>> > http://www.owasp.org/local/ireland.html
>>>>>>> > http://www.owasp.org/index.php/OWASP_Code_Review_Project
>>>>>>> > _______________________________________________
>>>>>>> > Owasp-testing mailing list
>>>>>>> > Owasp-testing at lists.owasp.org
>>>>>>> > https://lists.owasp.org/mailman/listinfo/owasp-testing
>>>>>>> >
>>>>>>> >
>>>>>>> _______________________________________________
>>>>>>> Owasp-testing mailing list
>>>>>>> Owasp-testing at lists.owasp.org
>>>>>>> https://lists.owasp.org/mailman/listinfo/owasp-testing
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Owasp-testing mailing list
>>>>>> Owasp-testing at lists.owasp.org
>>>>>> https://lists.owasp.org/mailman/listinfo/owasp-testing
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Owasp-testing mailing list
>>>> Owasp-testing at lists.owasp.org
>>>> https://lists.owasp.org/mailman/listinfo/owasp-testing
>>>>
>>>>
>>>
>>>
>>> --
>>> Eoin Keary OWASP - Ireland
>>> http://www.owasp.org/local/ireland.html
>>> http://www.owasp.org/index.php/OWASP_Code_Review_Project
>>>
>>> _______________________________________________
>>> Owasp-testing mailing list
>>> Owasp-testing at lists.owasp.org
>>> https://lists.owasp.org/mailman/listinfo/owasp-testing
>>>
>>>
>> _______________________________________________
>> Owasp-testing mailing list
>> Owasp-testing at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-testing
>>
>>
>>
>>
>> _______________________________________________
>> Owasp-testing mailing list
>> Owasp-testing at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-testing
>>
>>
>
> _______________________________________________
> Owasp-testing mailing list
> Owasp-testing at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-testing
>
>

-- 
Eoin Keary OWASP - Ireland
http://www.owasp.org/local/ireland.html
http://www.owasp.org/index.php/OWASP_Code_Review_Project
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.owasp.org/pipermail/owasp-testing/attachments/20080522/dbb1a31b/attachment-0001.html 