[Owasp-board] OWASP Website Speed

Matt Tesauro matt.tesauro at owasp.org
Wed Aug 21 12:55:56 UTC 2013

FYI:  The server was resized to a 8 GB version late last night - Texas time.

-- Matt's phone
On Aug 20, 2013 1:37 PM, "Matt Tesauro" <matt.tesauro at owasp.org> wrote:

> For the curious, I've attached the latest CPU utilization graph for the
> wiki server.  Same for the server load average.  The odd thing is that the
> spikes in both graphs do not directly follow our traffic patterns.  Its
> been difficult to figure out exactly what the cause is.  The upgrade to 8
> GB RAM server is a bit of a sledgehammer fix but should reduce those spikes
> to a more workable level.
> I've also setup monitoring alerts when CPU hits 90% and/or high load on
> that server so I get emails when either CPU or load is high and then a
> follow-on notification when the CPU/load returns to normal.
> Cheers!
> --
> -- Matt Tesauro
> OWASP WTE Project Lead
> http://www.owasp.org/index.php/Category:OWASP_Live_CD_Project
> http://AppSecLive.org - Community and Download site
> OWASP OpenStack Security Project Lead
> https://www.owasp.org/index.php/OWASP_OpenStack_Security_Project
> On Tue, Aug 20, 2013 at 11:47 AM, Matt Tesauro <matt.tesauro at owasp.org>wrote:
>> Sarah,
>> @ 3rd Party MediaWiki SaaS - Myself, Denis Cruz and some other OWASPers
>> looked into finding companies which provide supported MediaWiki installs.
>>  Besides cheap, shared hosting providers, we were unable to find anyone
>> that fit our needs/requirements.
>> @ Wiki change to 8 GB: Actually that change was one of the changes
>> scheduled for this weekend.  However, my daughter got sick Saturday night
>> and passed it on to me so that change didn't get done.  I was out Sunday
>> and Monday (from my life, day job and OWASP IT).  I've am planning to do
>> the VM resize up to 8 GB tonight since its the first low traffic time
>> available post me getting back to normal.
>> From chats with support Rackers, there are two methods to do that at Rack:
>> (1) Make an image of a running VM then use that image to create a new 8GB
>> server.  Then switch DNS to point at the new server's IP address.
>> (2) Directly resized a running VM up.  I don't like this option because:
>>    - If something goes wrong during the resize, there's no easy revert
>>    process.
>>    - The server will be rebooted during that process - it is automated
>>    and I cannot control when that happens.
>>    - DNS doesn't need to be switched but the reboot will cause down time.
>> For both, I will need to put the wiki in "Read-Only" mode to ensure there
>> are no content/DB changes during the resize process - which could cause
>> data loss.  Rough estimates on this are - rounding up with safety margins:
>>    30 minutes to make a VM image, 30 minutes to create a new server based
>> on that image, 10 minutes to update DNS.
>> For this full window, I will need the wiki in read-only.  After the DNS
>> change, I will keep the "smaller" 4 GB server in read-only mode to make
>> sure changes can only happen on the real server while DNS is propagating.
>> Considering all this, I don't want to do this during the middle of the
>> day (whatever that means for an international site).  I do know that
>> traffic is significantly reduced in the late evening in my time zone so I
>> figured I'd stay up a bit late tonight and make this change.
>> I should also note that OWASP let its IT infrastructure drift into its
>> current state over years of mild neglect.  The expectation that the
>> technical dept we built up over that time is going to quickly be retired
>> without great disruptions or time expenses is not valid.  Things will be
>> less than optimal for a while but we are continually progressing towards an
>> optimal end state.
>> Also, its is only one step in getting the wiki into a good state.  We
>> still have several issues effecting the wiki which will take a significant
>> amount of time to address:
>>    - Further review of extensions to remove those that are active but
>>    not used - may require setting the wiki in debug logging mode
>>       - Note many of the extensions we are using are either
>>       experimental, alpha, no longer supported or custom developed.  Any
>>       performance impacts of these extensions are unknown.
>>    - Update MediaWiki software to 1.20.x then 1.21.x so we are on the
>>    current branch.
>>    - Research intermittent high load on wiki server.  Have set
>>    monitoring warnings for high CPU, RAM and system load.  Some spikes appear
>>    to match increases in traffic.  Others are seem quite random - 4 AM Sunday
>>    morning, 2 PM Thursday, etc.  Spikes could be caused by
>>       - Update to Ubuntu 12-10
>>       - Update to MediaWiki 1.19.7
>>       - Update to MySQL 5.x
>>       - Update to PHP
>>       - Update to any of the extensions that are still active
>>       - One of the many beta and unsupported extensions which are
>>       currently active
>>       - One of the custom written plugins (by various OWASPers) which
>>       are currently active
>>       - It does not appear to be attack traffic from multiple log reviews
>>    - Depending on the research on the above, consider changing the
>>    architecture to better handle the wiki's work load
>> Does that help explain where things are at currently?
>> --
>> -- Matt Tesauro
>> OWASP WTE Project Lead
>> http://www.owasp.org/index.php/Category:OWASP_Live_CD_Project
>> http://AppSecLive.org - Community and Download site
>> OWASP OpenStack Security Project Lead
>> https://www.owasp.org/index.php/OWASP_OpenStack_Security_Project
>> On Tue, Aug 20, 2013 at 10:17 AM, Sarah Baso <sarah.baso at owasp.org>wrote:
>>> Matt -
>>> Could you give us an update on where you are with this?
>>> I know you made the following change last week: Increase the server to
>>> an 8 GB server from a 4 GB.  It will, not surprisingly, double the price
>>> from ~175.20/month to ~350.40/month but we'll also go from 2 CPUs to 4
>>> CPUs.  After the caching changes, it appears the bottle neck is CPU so that
>>> should really help.  There is still a big issue with the load time
>>> (although some improvement from when we last talked).
>>> Thanks,
>>> Sarah
>>> On Tue, Aug 20, 2013 at 7:14 AM, Michael Coates <
>>> michael.coates at owasp.org> wrote:
>>>> Sarah,
>>>> I know you're looking into the current owasp website speed issue. Do
>>>> let us know what you find out.
>>>> Also, can I request that you take a look at options for third party
>>>> services for wiki hosting & management. I have to believe there are groups
>>>> that would host and run a wiki site. We should see if this makes sense for
>>>> us or not.
>>>> Thanks!
>>>> --
>>>> Michael Coates | OWASP | @_mwc
>>> --
>>> Executive Director
>>> OWASP Foundation
>>> sarah.baso at owasp.org
>>> +1.312.869.2779
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20130821/de2216b0/attachment.html>

More information about the Owasp-board mailing list