[OWASP-WEBSCARAB] Using WebScarab as a Web Capture program...
rogan at dawes.za.net
Thu Jan 5 00:47:35 EST 2006
David Stidolph wrote:
> Hello, my name is David Stidolph and I was tasked with building a Proxy
> Server that can capture Web Pages and play them back in the future when
> the web server is no longer available. I wrote a Tomcat Servlet that
> does this for HTTP sites, but it does NOT work for HTTPS sites.
Ok, before we go off the beaten track, lets take a look at what you
would need to do to get your Tomcat servlet to work.
The basic problem is that when the browser tries to connect to an HTTPS
server via your tomcat servlet/proxy (I assume that you configure your
browser to use your servlet as a proxy in some way?), it issues a
request that looks like:
CONNECT targethost:port HTTP/1.0\r\n
[optional header lines\r\n]
It then expects the proxy to give a "200 Ok" response, and connect it
directly to the target port, where it can negotiate an SSL connection
with the real web server. If this happens, your servlet will not be able
to intercept to modify or respond to the requests.
What WebScarab (and any other intercepting proxy that supports HTTPS)
does is issue a 200 Ok response, but NOT connect the browser to the
server. It then tries to negotiate an SSL connection with the browser
directly, so that the browser is talking to WebScarab, rather than the
actual web server. WebScarab can then do whatever it wants to with the
I don't know any servlet programming, so I don't know how hard it would
be for you to implement this. You might want to take a look at the code
in ConnectionHandler to see how the SSL connection is negotiated.
> WebScarab looks like a way to get this done for me.
> What I need is for the Proxy Server to work in three modes:
> Passthrough, Record and Playback.
> Normally the mode would be Passthrough so files would pass through
> without changes or even monitoring. Recording would be turned on and
> headers/files would be cached until recording is ended and the files are
> written into a zip file. Playback would load the zip file and as each
> file request comes in it would respond with the file/headers.
> What I have now is a set of classes. One class wraps up a file (all
> files - HTML, GIF, CSS, etc) contents, headers and URL. Another class
> is a Table Of Contents (TOC) that tracks individual files (everything
> from HTML to JS to CSS to GIF, etc). This code is working as a servlet,
> but I would have to change it to use the WebScarab framework.
> Rogan Dawes and I have talked and he suggested this might be possible as
> a Bean Shell script.
> I can certainly see how I can record files, but I will also have to
> block going to the internet for files and provide them myself (for
> playback and responding to change of state requests like recording to a
> zip file).
> Even though we will NOT be shipping this code outside the company I work
> for, I have gotten permission to share it and contribute it back to the
> Ok, given this, what are my questions???
> #1 I see the sample Bean Shell script. This is real java? I can add a
> jar file I build outside of it and call methods in it?
Yes, you can. Take a look at www.beanshell.org. The only possibly tricky
part would be making sure that your classpath is properly set up, so
that the classes can be found.
> #2 If I do NOT call nextPlugin.fetchResponse can I construct my own
> Response object to go back to the browser (for playback and responding
> to commands)?
> #3 How about debugging? Will System.out.println commands go to the
> command window or can I use the Java logging commands?
System.out.println will go to the console (command window from which you
launched WebScarab), or you can use the java.util.logging.Logger
classes. WebScarab copies any messages in the org.owasp.webscarab
hierarchy to the Message Log pane in WebScarab as well.
> Thanks for any answers and feedback.
> David Stidolph
> P.S. Please don't assume I know much - this is my second Java program.
> Mostly I know C++ Windows programming...be gentle.
More information about the Owasp-webscarab