[Owasp-webscarab] RE: WebScarab

Wed Jul 16 04:46:13 EDT 2003

Hi Rohit, Ingo.

I have also been fairly quiet on the Scarab front, so don't feel too bad.

There are a couple of things that have happened, though, and you may want to
check the archives of this list for my last two or so messages.

I have put up an "architecture diagram" on my website
(http://mysite.mweb.co.za/residents/rdawes/Exodus_Arch.png
<http://mysite.mweb.co.za/residents/rdawes/Exodus_Arch.png> ), as well as a
very early code snapshot at
http://mysite.mweb.co.za/residents/rdawes/webscarab.tar.gz
<http://mysite.mweb.co.za/residents/rdawes/webscarab.tar.gz> )

The current proposal for WebScarab is as follows:

We differentiate between different kinds of plugins.

WebScarab itself is the "container" for the plugins. WebScarab plugins have
two modes of operation: Firstly, they generate requests, collect the
responses, and submit those conversations to WebScarab for processing. (My
current implementation has the conversations submitted to the overall model,
which I think I should change). Secondly, the plugin can provide additional
analysis of responses, in terms of setting properties of the conversation. 

The overall site model is responsible for summarising and collating the
individual conversation properties, and presenting any conclusions that are
obtained.

An example of a WebScarab plugin would be a spider, the proxy, the Exodus
fuzzer, etc.

Then there are proxy plugins:

Proxy plugins are useful in that they allow for the behaviour of the browser
to be altered dynamically, for example, setting headers, adding or removing
cookies, changing parameter values, etc. But these proxy plugins have no
necessity to perform any analysis. They are roughly equivalent to a
"FilterInputStream", if you like.

Finally, there are interface plugins, which allow for interaction with the
user in some way or another. The definition of the Interface plugin will
likely change depending on the nature of the interface selected. For
example, a Swing plugin would look somewhat different to a plugin into a web
interface, etc.

I have tried to separate the user interface from the WebScarab plugins as
far as possible. Any suggestions on how best to achieve this would be
appreciated.

Going forward, I intend to move parts of the existing WebScarab spider
module into a higher level, so it can be shared by all. The page
classifiers, definitions of a Link, the HTML parser, should all be shared in
my opinion. 

The spider plugin becomes simpler. As all WebScarab Plugins do, it provides
a method to analyse a response. When it is provided with a response, it
simply takes any Links in that response parsed by the overall model, and
identifies whether it has been seen yet or not. The user interface provides
a method of selecting which of those Links to fetch (selecting from a table,
excluding those that match a regex, etc), and calls the spider plugin to
fetch those Links. The Spider plugin would add request headers, add any
Cookies that are appropriate, thread the requests, and send the responses
into the model, perpetuating the cycle.

The Proxy plugin, which is partially implemented already in the tarball,
needs a little more fleshing out, particularly in terms of synchronising
Cookies and other Session State between the Proxy and the Spider, but
otherwise is substantially done. Maybe some reliability enhancements would
be good, and possibly support for Konqueror's method of requesting SSL
connections, but not much besides. Now effort needs to go into making useful
proxy plugins, and the user interfaces for those.

Maybe you'd like to take a stab at some of the proxy plugins, Rohit? 

I  have basically got a "ManualEdit" plugin complete, but I have had
requests for something that allows you to specify a regular expression, that
would result in a match having that regular expression effected on the
request or response. I'm thinking of something along the lines of Perl's
's/Host: (.*)\.com/Host: ($1)\.co.za/g' for example.

Other work that needs doing is deciding how the overall model will interact
with the conversation model, in terms of summarising data points.

For example, we may have many requests for a particular URL. Some of them
are observed by a plugin to have XSS vulnerabilities, some are observed to
show signs of ODBC errors, etc. We would like to be able to summarise that,
to indicate that the URL has XSS vulnerabilities, as well as SQL injection
vulnerabilities, ideally to a depth of which parameter is affected. Other
things might be HTML comments that should be summarised (only show a
particular fragment once, even if it appears in every single instance of the
response), and script fragments. Or determining whether a particular URL
varies based on the presence or absence of a cookie or other session
identifier.

Rogan

-----Original Message-----
From: S. Rohit [mailto:s.rohit at usa.net] 
Sent: 15 July 2003 06:06 PM
To: ingo at ingostruck.de; Dawes, Rogan (ZA - Johannesburg)
Subject: WebScarab

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

hi guyz...

    sorry for no repleis or activity on my part. been out of town all this
while on a client project. will try to make a more active contribution now.
ingo i remember u planning a code walk through. is tat already done or still
in the pipeline? cos i ealize that every time im only playing catch up to
all ur emails rather than making any real contribs. this is cos im juz not
able to take out enuf time to sit thru and go thru the entire code. also
i've ralised that mebbe if u assign some specific task for me first, so that
i can focus on it then i can start contributing slowly tat way. i mite be
very irrgualr and missing deadliens again but atleast will eb doing
something. sorry abt all this lack of discipline guys. hv been trying hard
but juz not able tot ake time out.

rohit

p.s. rogan is there any gud help documentation or stuff for exodus. was
trying in office for a client project. seems pretty ahrd to figure out some
stuff.
-----BEGIN PGP SIGNATURE-----
Version: PGP 8.0

iQIVAwUBPxQmM18Z3uKHDtA5AQLEzBAAqY6td//1crhVYddxGCkwvuiNiWpQkRcu
48oMjTQUllOF+TfSIx8EDIk3mZQJstjZczIUPVwHbJ0/mODds15EHx3OZXLa3qEq
bZ9RilhcD2LTDSxxtX/hHC+ob9am3cBZ+b/Bxaontbx9S1U15E5qMxyVRIMvc/dD
XPV3/gO8TgQIN8L4ZhAd2Ok4HHOUtmmGVf3uX/p9N4GNyXCKuzWxhSLiMDOfAqjR
sWQAX2aV59KwnuPL8lC+KcwbFx/TClMGOTwIQdvN5MhWKdE17Ucoedx+tcCSzVSi
m2cYh4q4snOmVNMB7tAu/EXopKOmJILbVYrxFJ3jKs/28MX6xLa3+4iqFLzX0R1z
saEJqDX12xAFM6BB9JIqd7K7N9uR+zAtWJpSo9MVG2gSYHGhodZjsBFSlc38d5/c
DGJGn5tnOFtSwO13pnXtUWcDMU0dRimRyPLHGhhm9OT6VDKQ0HWIrxx+8bK3SvBA
PTKnjgjdoNP4ly2Opl/HEqDfjSBCpVaPEWr/ev/bTSqQfRh9Ozu/VlAQIH1p7MB6
pP8nUt/8787aoJ6mq8pEThmTjzuoH8M6LbtfpyxwfQqWdBqah63xdRH9THMo5pEV
e4556m2ISRM+ozSwR2/XDLwjbQ05vPz02fNR7dQ3y3iC6zjl/OBxB932icg2A0VP
4R4cuPyPF/g=
=k6C8
-----END PGP SIGNATURE-----

Important Notice: This email is subject to important restrictions, qualifications and disclaimers ("the Disclaimer") that must be accessed and read by clicking here or by copying and pasting the following address into your Internet browser's address bar: http://www.Deloitte.co.za/Disc.htm. The Disclaimer is deemed to form part of the content of this email in terms of Section 11 of the Electronic Communications and Transactions Act, 25 of 2002. If you cannot access the Disclaimer, please obtain a copy thereof from us by sending an email to ClientServiceCentre at Deloitte.co.za.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.owasp.org/pipermail/owasp-webscarab/attachments/20030716/116e6ecb/attachment.html 