[Owasp-dirbuster] Performance HOWTO

[Andres Riancho]

James,

On Tue, Oct 28, 2008 at 9:42 AM, James Fisher
<dirbuster at sittinglittleduck.com> wrote:
>
> Oops forgot to replay to the list....
>
> Quoting Andres Riancho <andres.riancho at gmail.com>:
>
>> James,
>>
>>     Hi! How are you? I hope you are well.
>
> Good, thanks, been working on my other project FuzzBuster of late,
> which I hope to give a public release soon :)

Something in the lines of JBroFuzz?

>>
>>     During last week I've been trying to enhance the performance of
>> w3af, and particularly the performance of the HTTP requests. After
>> some thinking on how to achieve high speeds, I remembered that
>> DirBuster does an AWESOME job at this, so I wanted to ask you some
>> questions related to the way you used to measure the performance of
>> your tool:
>>
>> - Have you used any specialized tools? Which?
>
> In terms of testing performance, the only specialized tool, would be
> profiler within netbeans.  That enables me to see what each thread is
> up to and fine tune the threading.
>
>> - Do you perform only HEAD requests?
>
> I try to, but it's not always possible.  The bigger factor is make use
> you are using keep-alives.  If not then especially on a windows
> machine you will run out of local ports to use for connections.

I'm using keep-alive, so I'm on the right path I think.... =)

>> - Do you have the test server (apache?) running on localhost?
>
> pretty much, with a couple of tweaks to make it return 200's for 404,
> another such things.  Plus is gets testing on lots of live servers as
> part of my work

ok

>> - When running on local host, in which section are you having the bottleneck?
>
> Resource on the machine will max out, I think I haven't performed that
> test for a while.

hmmm... nice... I haven't really got there. I only have 400 GET
requests/second with one thread.

>> - When running on a network... is the network link the bottleneck?
>
> Depends on the network, it can be, this is were head requests work well

ok

>> - For your tests, do you fine tune with a single thread, and then
>> simply run different threads?
>
> If it's performance I'm testing then, I wont run it on a single
> thread.  But being able to switch down to a single thread is really
> useful for debugging.

Yes, threads are evil.

>
>> - I've read somewhere that threads are evil, and that you should avoid
>> them when they compete for a scarce resource like network. What do you
>> think about this?
>
> If you want speed, then you have to use threads, you just have to get
> the time each thread is designed to sleep for right.  Too much or too
> little sleep will slow the program down.

Sleep? My threads just run and the thread management is handled by
python. I don't force any thread to sleep at any time. The "sleep" is
in socket.recv()

> Most of dirbuster is all threads.  The workers, work generators, html
> parsers, performance monitors and there is also a thread to prevent
> deadlock.

I HAVE to read that code!

>> - How long do you run each test? 1, 2, 5 minutes?
>
> Till the program finishes, but I run it on a server, that I know
> DirBuster will finish is a sensible time 5 - 10

In my tests I'm sending 3000 to 6000 GET requests to the same URL, and
I analyze the time with a little python script that reads the apache
log. I've found that 3000 requests is more than enough to see how fast
requests are being sent.

>> - How confident are you about the results you get from your
>> tools/testing methodology?
>
> In terms of a formal methodology testing methodology I don't really
> have one.  But DirBuster has had 3-4 years of field testing on live
> systems.  So over time I have managed to iron out most of the bugs,
> and spot areas where I can improve the performance and it's reliability.

w3af has 2-3 years, but I haven't really focused on performance
before... all this time has been: "make things work, fix critical
bugs, remove false positives". Now that we finished with that, we can
focus a little in performance, which is *really* hard!!

>>
>>     I'm sorry if this seems an interrogatory, but if you are as
>> passionate as I am about this... I think you'll rather enjoy this
>> thread =)
>
> No problems, if you have more question please feel free to ask!

Thanks for your answers!

> James
>
>>
>> Cheers,
>> --
>> Andres Riancho
>> http://w3af.sourceforge.net/
>> Web Application Attack and Audit Framework
>> _______________________________________________
>> Owasp-dirbuster mailing list
>> Owasp-dirbuster at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-dirbuster
>>
>
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>
>
> ----- End forwarded message -----
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
> _______________________________________________
> Owasp-dirbuster mailing list
> Owasp-dirbuster at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-dirbuster
>



-- 
Andres Riancho
http://w3af.sourceforge.net/
Web Application Attack and Audit Framework