[Owasp-board] OWASP Website Speed

Matt Tesauro matt.tesauro at owasp.org
Tue Aug 20 18:37:01 UTC 2013


For the curious, I've attached the latest CPU utilization graph for the
wiki server.  Same for the server load average.  The odd thing is that the
spikes in both graphs do not directly follow our traffic patterns.  Its
been difficult to figure out exactly what the cause is.  The upgrade to 8
GB RAM server is a bit of a sledgehammer fix but should reduce those spikes
to a more workable level.

I've also setup monitoring alerts when CPU hits 90% and/or high load on
that server so I get emails when either CPU or load is high and then a
follow-on notification when the CPU/load returns to normal.

Cheers!

--
-- Matt Tesauro
OWASP WTE Project Lead
http://www.owasp.org/index.php/Category:OWASP_Live_CD_Project
http://AppSecLive.org - Community and Download site
OWASP OpenStack Security Project Lead
https://www.owasp.org/index.php/OWASP_OpenStack_Security_Project


On Tue, Aug 20, 2013 at 11:47 AM, Matt Tesauro <matt.tesauro at owasp.org>wrote:

> Sarah,
>
> @ 3rd Party MediaWiki SaaS - Myself, Denis Cruz and some other OWASPers
> looked into finding companies which provide supported MediaWiki installs.
>  Besides cheap, shared hosting providers, we were unable to find anyone
> that fit our needs/requirements.
>
> @ Wiki change to 8 GB: Actually that change was one of the changes
> scheduled for this weekend.  However, my daughter got sick Saturday night
> and passed it on to me so that change didn't get done.  I was out Sunday
> and Monday (from my life, day job and OWASP IT).  I've am planning to do
> the VM resize up to 8 GB tonight since its the first low traffic time
> available post me getting back to normal.
>
> From chats with support Rackers, there are two methods to do that at Rack:
>
> (1) Make an image of a running VM then use that image to create a new 8GB
> server.  Then switch DNS to point at the new server's IP address.
>
> (2) Directly resized a running VM up.  I don't like this option because:
>
>    - If something goes wrong during the resize, there's no easy revert
>    process.
>    - The server will be rebooted during that process - it is automated
>    and I cannot control when that happens.
>    - DNS doesn't need to be switched but the reboot will cause down time.
>
> For both, I will need to put the wiki in "Read-Only" mode to ensure there
> are no content/DB changes during the resize process - which could cause
> data loss.  Rough estimates on this are - rounding up with safety margins:
>    30 minutes to make a VM image, 30 minutes to create a new server based
> on that image, 10 minutes to update DNS.
>
> For this full window, I will need the wiki in read-only.  After the DNS
> change, I will keep the "smaller" 4 GB server in read-only mode to make
> sure changes can only happen on the real server while DNS is propagating.
>
> Considering all this, I don't want to do this during the middle of the day
> (whatever that means for an international site).  I do know that traffic is
> significantly reduced in the late evening in my time zone so I figured I'd
> stay up a bit late tonight and make this change.
>
> I should also note that OWASP let its IT infrastructure drift into its
> current state over years of mild neglect.  The expectation that the
> technical dept we built up over that time is going to quickly be retired
> without great disruptions or time expenses is not valid.  Things will be
> less than optimal for a while but we are continually progressing towards an
> optimal end state.
>
> Also, its is only one step in getting the wiki into a good state.  We
> still have several issues effecting the wiki which will take a significant
> amount of time to address:
>
>    - Further review of extensions to remove those that are active but not
>    used - may require setting the wiki in debug logging mode
>       - Note many of the extensions we are using are either experimental,
>       alpha, no longer supported or custom developed.  Any performance impacts of
>       these extensions are unknown.
>    - Update MediaWiki software to 1.20.x then 1.21.x so we are on the
>    current branch.
>    - Research intermittent high load on wiki server.  Have set monitoring
>    warnings for high CPU, RAM and system load.  Some spikes appear to match
>    increases in traffic.  Others are seem quite random - 4 AM Sunday morning,
>    2 PM Thursday, etc.  Spikes could be caused by
>       - Update to Ubuntu 12-10
>       - Update to MediaWiki 1.19.7
>       - Update to MySQL 5.x
>       - Update to PHP
>       - Update to any of the extensions that are still active
>       - One of the many beta and unsupported extensions which are
>       currently active
>       - One of the custom written plugins (by various OWASPers) which are
>       currently active
>       - It does not appear to be attack traffic from multiple log reviews
>    - Depending on the research on the above, consider changing the
>    architecture to better handle the wiki's work load
>
> Does that help explain where things are at currently?
>
>
> --
> -- Matt Tesauro
> OWASP WTE Project Lead
> http://www.owasp.org/index.php/Category:OWASP_Live_CD_Project
> http://AppSecLive.org - Community and Download site
> OWASP OpenStack Security Project Lead
> https://www.owasp.org/index.php/OWASP_OpenStack_Security_Project
>
>
> On Tue, Aug 20, 2013 at 10:17 AM, Sarah Baso <sarah.baso at owasp.org> wrote:
>
>> Matt -
>> Could you give us an update on where you are with this?
>>
>> I know you made the following change last week: Increase the server to
>> an 8 GB server from a 4 GB.  It will, not surprisingly, double the price
>> from ~175.20/month to ~350.40/month but we'll also go from 2 CPUs to 4
>> CPUs.  After the caching changes, it appears the bottle neck is CPU so that
>> should really help.  There is still a big issue with the load time
>> (although some improvement from when we last talked).
>>
>> Thanks,
>> Sarah
>>
>>
>> On Tue, Aug 20, 2013 at 7:14 AM, Michael Coates <michael.coates at owasp.org
>> > wrote:
>>
>>> Sarah,
>>>
>>> I know you're looking into the current owasp website speed issue. Do let
>>> us know what you find out.
>>>
>>> Also, can I request that you take a look at options for third party
>>> services for wiki hosting & management. I have to believe there are groups
>>> that would host and run a wiki site. We should see if this makes sense for
>>> us or not.
>>>
>>> Thanks!
>>>
>>> --
>>> Michael Coates | OWASP | @_mwc
>>>
>>>
>>
>>
>> --
>> Executive Director
>> OWASP Foundation
>>
>> sarah.baso at owasp.org
>> +1.312.869.2779
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20130820/c0902a06/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wiki-cpu-utilization-graph.png
Type: image/png
Size: 37734 bytes
Desc: not available
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20130820/c0902a06/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wiki-server-load-graph.png
Type: image/png
Size: 32432 bytes
Desc: not available
URL: <http://lists.owasp.org/pipermail/owasp-board/attachments/20130820/c0902a06/attachment-0003.png>


More information about the Owasp-board mailing list