[Owasp-webscarab] RE: WebScarab
Dawes, Rogan (ZA - Johannesburg)
rdawes at deloitte.co.za
Wed Jul 16 04:46:13 EDT 2003
Hi Rohit, Ingo.
I have also been fairly quiet on the Scarab front, so don't feel too bad.
There are a couple of things that have happened, though, and you may want to
check the archives of this list for my last two or so messages.
I have put up an "architecture diagram" on my website
<http://mysite.mweb.co.za/residents/rdawes/Exodus_Arch.png> ), as well as a
very early code snapshot at
The current proposal for WebScarab is as follows:
We differentiate between different kinds of plugins.
WebScarab itself is the "container" for the plugins. WebScarab plugins have
two modes of operation: Firstly, they generate requests, collect the
responses, and submit those conversations to WebScarab for processing. (My
current implementation has the conversations submitted to the overall model,
which I think I should change). Secondly, the plugin can provide additional
analysis of responses, in terms of setting properties of the conversation.
The overall site model is responsible for summarising and collating the
individual conversation properties, and presenting any conclusions that are
An example of a WebScarab plugin would be a spider, the proxy, the Exodus
Then there are proxy plugins:
Proxy plugins are useful in that they allow for the behaviour of the browser
to be altered dynamically, for example, setting headers, adding or removing
cookies, changing parameter values, etc. But these proxy plugins have no
necessity to perform any analysis. They are roughly equivalent to a
"FilterInputStream", if you like.
Finally, there are interface plugins, which allow for interaction with the
user in some way or another. The definition of the Interface plugin will
likely change depending on the nature of the interface selected. For
example, a Swing plugin would look somewhat different to a plugin into a web
I have tried to separate the user interface from the WebScarab plugins as
far as possible. Any suggestions on how best to achieve this would be
Going forward, I intend to move parts of the existing WebScarab spider
module into a higher level, so it can be shared by all. The page
classifiers, definitions of a Link, the HTML parser, should all be shared in
The spider plugin becomes simpler. As all WebScarab Plugins do, it provides
a method to analyse a response. When it is provided with a response, it
simply takes any Links in that response parsed by the overall model, and
identifies whether it has been seen yet or not. The user interface provides
a method of selecting which of those Links to fetch (selecting from a table,
excluding those that match a regex, etc), and calls the spider plugin to
fetch those Links. The Spider plugin would add request headers, add any
Cookies that are appropriate, thread the requests, and send the responses
into the model, perpetuating the cycle.
The Proxy plugin, which is partially implemented already in the tarball,
needs a little more fleshing out, particularly in terms of synchronising
Cookies and other Session State between the Proxy and the Spider, but
otherwise is substantially done. Maybe some reliability enhancements would
be good, and possibly support for Konqueror's method of requesting SSL
connections, but not much besides. Now effort needs to go into making useful
proxy plugins, and the user interfaces for those.
Maybe you'd like to take a stab at some of the proxy plugins, Rohit?
I have basically got a "ManualEdit" plugin complete, but I have had
requests for something that allows you to specify a regular expression, that
would result in a match having that regular expression effected on the
request or response. I'm thinking of something along the lines of Perl's
's/Host: (.*)\.com/Host: ($1)\.co.za/g' for example.
Other work that needs doing is deciding how the overall model will interact
with the conversation model, in terms of summarising data points.
For example, we may have many requests for a particular URL. Some of them
are observed by a plugin to have XSS vulnerabilities, some are observed to
show signs of ODBC errors, etc. We would like to be able to summarise that,
to indicate that the URL has XSS vulnerabilities, as well as SQL injection
vulnerabilities, ideally to a depth of which parameter is affected. Other
things might be HTML comments that should be summarised (only show a
particular fragment once, even if it appears in every single instance of the
response), and script fragments. Or determining whether a particular URL
varies based on the presence or absence of a cookie or other session
From: S. Rohit [mailto:s.rohit at usa.net]
Sent: 15 July 2003 06:06 PM
To: ingo at ingostruck.de; Dawes, Rogan (ZA - Johannesburg)
-----BEGIN PGP SIGNED MESSAGE-----
sorry for no repleis or activity on my part. been out of town all this
while on a client project. will try to make a more active contribution now.
ingo i remember u planning a code walk through. is tat already done or still
in the pipeline? cos i ealize that every time im only playing catch up to
all ur emails rather than making any real contribs. this is cos im juz not
able to take out enuf time to sit thru and go thru the entire code. also
i've ralised that mebbe if u assign some specific task for me first, so that
i can focus on it then i can start contributing slowly tat way. i mite be
very irrgualr and missing deadliens again but atleast will eb doing
something. sorry abt all this lack of discipline guys. hv been trying hard
but juz not able tot ake time out.
p.s. rogan is there any gud help documentation or stuff for exodus. was
trying in office for a client project. seems pretty ahrd to figure out some
-----BEGIN PGP SIGNATURE-----
Version: PGP 8.0
-----END PGP SIGNATURE-----
Important Notice: This email is subject to important restrictions, qualifications and disclaimers ("the Disclaimer") that must be accessed and read by clicking here or by copying and pasting the following address into your Internet browser's address bar: http://www.Deloitte.co.za/Disc.htm. The Disclaimer is deemed to form part of the content of this email in terms of Section 11 of the Electronic Communications and Transactions Act, 25 of 2002. If you cannot access the Disclaimer, please obtain a copy thereof from us by sending an email to ClientServiceCentre at Deloitte.co.za.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Owasp-webscarab