[OWASP-WEBSCARAB] new release of Webscarab

[Rogan Dawes]

Hi folks,

I have made a new release of WebScarab available on Sourceforge, to 
celebrate it being placed in position 35 of Fyodor's annual-ish pen 
testing tools survey.

You can download it here:

<http://sourceforge.net/project/showfiles.php?group_id=64424&package_id=61823>

There are quite a few smallish changes under the hood, which you can see 
in the associated Changelog. I'll summarise the more interesting ones here:

BugFixes
========

The Parsed Request Editor would corrupt headers/parameters where the 
name contained url-encoded characters. We would url-decode the name to 
show it in the more user-friendly manner, but we didn't reencode it 
before sending it to the server.

Enhancements
============

WebScarab now has a logo! ;-) Thanks to Andrew van der Stock and Larry 
Casey for their assistance in getting this done.

Probably the most visible change is a new Memory utilisation widget that 
runs across the bottom of the UI. This allows users to monitor their 
memory utilisation, and save their session prior to running out of 
memory. It will also change color from green to yellow to red when one 
approaches the limit. (Yes, I know WebScarab has some pretty serious 
memory leaks)

There is a new Lite version of WebScarab, which only shows the Summary 
page, and the Proxy Intercept tab. Some folks have commented that 
WebScarab is intimidating to new users, with all the options. This 
interface is intended to address those concerns. Note that it is not the 
default, and needs to be invoked as "java -jar webscarab.jar Lite", in 
order to be activated.

A related change is a hidden option to discard any conversations where 
the URL matches a user-supplied regular expression. Again, for new 
users, things like images, javascripts(?), Cascading Style Sheets, etc 
simply clutter the the summary, and are confusing. By adding a property 
to the webscarab.properties file in the user.home directory, one can 
enable this feature. For example:

WebScarab.dropRegex=.*\\.(gif|jpg|png|css)$

There is a hidden option to change the layout of the Conversation 
windows. Previously I polled list members to get their preferences, 
either Request above Response, or Request next to Response. I got 
support for both variations, so I implemented a hot-key to switch 
between them, namely Ctrl-T.

The underlying HTTP engine has been reworked to support authentication 
internally, rather than punting and letting the browser handle it. 
WebScarab now maintains a set of credentials, and uses them 
automatically when appropriate. This means that WebScarab can now 
support NTLM-authentication, thanks to code taken from the jCIFS project.

The Summary now defaults to sorting in reverse. This means that the most 
recent conversations appear at the top of the table, instead of at the 
bottom. It is easy to change this (non-permanently, unfortunately), by 
clicking on the column headers to choose your preferred sort order.

The Text Pane in the conversation view now has reworked Search 
capabilities. Now if you press Ctrl-F, you get a FireFox-like search 
bar, that highlights all matches for your search as you type it. Most 
interestingly, it also supports regular expressions, which means that 
you can find an unknown piece of text between two known parts. It will 
also automatically scroll the text pane to show the first match if it is 
not on screen. This is best used with a regex group that matches the 
text that you are interested in, as only the matching group/s will be 
highlighted.

SSL Client Certificate management has also been reworked. WebScarab now 
supports using SSL Client certificates from PKCS#11 devices (please let 
me know if you have any trouble with this), as well as from PKCS#12 files.

The HttpUrl class has been reworked to not accept whitespace in URL's. 
This was important, because parsing malformed HTML might result in 
corruption of the user's session, preventing later reloading of the session.

Some minor tweaks were made to the HTML pane, so that images would not 
be shown. Previously, images would be loaded from the originating 
server, if the HTML included a BASE parameter, or if the URL was absolute.

Manual Intercept:

The buttons have been moved around a bit, to make them a bit more 
obvious/intuitive, hopefully.

Plugins
=======

Spider plugin:

Now understands relative paths in Location headers. (I don't think this 
is according to the spec, but it is common enough to ned

SessionID Plugin:

You can now clear a list of sessionids, if you want to take a 
new/different sample.

You can also export the currently selected samples, for external 
manipulation in e.g. a spreadsheet, etc.

It is also possible to extract more than one piece of a sessionid at one 
time, by using multiple groups. The previous behaviour was simply to 
concatenate each of the matching groups into one long string, and 
analyse that. Now we capture the original session id, as well as new 
sessionids that are made up only of the matching groups. In other words, 
you might expect to see "localhost/ JSESSIONID", "localhost/ JSESSIONID 
1", "localhost/ JSESSIONID 2", etc, if you used a regular expression 
like ..(....)..(....)..* to capture characters 3-6 and 9-12.

There are also statistics reported from the calculated values, showing 
the smallest value, the largest value, and the scale of the numbers in 
scientific notation.

Extension Checking:

There is a new plugin, contributed by Meder Kydyraliev, which allows the 
operator to selectively check URL's for variants by adding an extension. 
For example, by adding .bak, .old, ~, etc, one may be able to find older 
versions of pages that would not be executed on the server, but returned 
in clear to the browser. You can also check directories for archives 
from which they might have been created. e.g. myapp/ might become 
myapp.zip, myapp.tar, myapp.tar.gz, etc.

Fuzzer plugin:

The fuzzer plugin now has a table of conversations collected during the 
most recent run. You view the conversation details by double-clicking on 
a row, as one would expect, and step forward and backwards through the 
table. This makes it a lot easier to review the results.

The fuzzer now supports a new type of fuzz source, generated from a 
reduced regular expression. By reduced, I mean that you can't use . (any 
character), and * (variable numbers of characters). However, you can use 
ranges [A-Z], counts {3}, etc. WebScarab will then generate a list of 
all strings that match the regular expression.

As always feedback is welcome, preferably on the owasp-webscarab list at 
lists.sourceforge.net.

Thanks

Rogan