[Owasp-proxy] Defragmenting tcp packets

Martin Holst Swende martin at swende.se
Sun May 22 15:43:55 EDT 2011


On 05/22/2011 09:10 PM, Rogan Dawes wrote:
> On 2011/05/22 8:41 PM, Martin Holst Swende wrote:
>> On 04/03/2011 10:27 PM, Rogan Dawes wrote:
>>> On 2011/04/03 10:12 PM, Martin Holst Swende wrote:
>>>> Hi Rogan et al,
>>>>
>>>> Some time ago we discussed defragmentation of intercepted tcp packets.
>>>> Just wanted to make a 'ping' about this proposed patched version of
>>>> run() for the RelayInterceptor. It would be really nice to get it in
>>>> there, so I can use a 'vanilla' version of owasp-proxy instead of a
>>>> patched one.
>>>> Regards,
>>>> Martin Swende
>>>>
>>>> Alternative diff-version:
>>>> ---------------------------------
>>>>
>>>> --- a/src/main/java/org/owasp/proxy/tcp/RelayInterceptor.java
>>>> +++ b/src/main/java/org/owasp/proxy/tcp/RelayInterceptor.java
>>>> @@ -1,5 +1,6 @@
>>>>  package org.owasp.proxy.tcp;
>>>>  
>>>> +import java.io.ByteArrayOutputStream;
>>>>  import java.io.IOException;
>>>>  import java.io.InputStream;
>>>>  import java.io.OutputStream;
>>>> @@ -36,10 +37,30 @@ class RelayInterceptor<C, S> extends Thread
>>>> implements Strea
>>>>  
>>>>         public void run() {
>>>>                 try {
>>>> -                       byte[] buff = new byte[4096];
>>>> +                       ByteArrayOutputStream baos = new
>>>> ByteArrayOutputStream(4
>>>> +                       byte[] readBuffer = new byte[4096];
>>>> +                       byte[] finalBuffer;
>>>>                         int got;
>>>> -                       while ((got = in.read(buff)) > -1 && !done) {
>>>> -                               interceptor.received(this, buff, 0, got);
>>>> +                       while ((got = in.read(readBuffer)) > -1 && !done) {
>>>> +                               baos.write(readBuffer,0,got);
>>>> +                               try {
>>>> +                                       Thread.sleep(5);
>>>> +                               } catch (InterruptedException e) {
>>>> +                                       e.printStackTrace();
>>>> +                               }
>>> Have you tried it without the sleep(5)?
>>>
>>> What happens?
>>>
>> Hi, and sorry for the looong delay before this answer. I wanted to
>> verify exactly what was happening, which took some code-reverting,
>> rebuilding and getting time for wiresharktrapping and looking at it. As
>> I suspected, you are completely right - the reason for the fragmentation
>> is that it corresponds to the underlying tcp packets.
>>
>>> It seems to me that if there are bytes available, it must be because
>>> they arrived in the same packet as the one we just read from. In which
>>> case, in.available() should return > 0 immediately, and if it doesn't,
>>> then the data was not sent in the same packet, and should not be
>>> defragmented.
>> I am a bit torn here. Ostensibly,  you are correct. However, tcp
>> interception in java is not really a tool for actually doing operations
>> on the tcp level, only on the application level. For example, we cannot
>> do what nmap does, and send bogus status flags or syn-packets and such
>> things. A tcp interceptor written in Java can never guarantee (?) that
>> one packet in equals one packet out.
>>
>> So I'm thinking that a typical user will want to intercept and modify
>> the layer above tcp, and would probably benefit from tcp
>> defragmentation. The most efficient way to provide that would be if the
>> library could provide an optional 'delayed read'-interception which I
>> suggested and by letting the application layer decide which to use. If
>> an application does it's own defragmentation it means a whole lot of
>> unnecessary object construction/destruction and byte shuffling. So, do
>> you think we could somehow add such support, which by default would be off?
> True, I agree that the typical user would want to see the defragmented
> stream. But would it not make sense to allow a higher layer to perform
> the defragmentation if desired, rather than losing that information
> entirely with no way of getting it back?
That would definitely be the nicest thing to do. However, when I capture
some http traffic (for testing) on a
media rich site, FF opens up ~6 connections and pipelines data over each
one. All of a sudden, there are 300 hundreds of packets waiting in the
queue, I'm thinking that keeping that information and performing
de/re-fragmentation on the fly (maintaining original packets and mapping
that to a 'merged-packet-queue' all the while more packets may be
arriving) may be problematic.

For now, I at least would settle for a tick-the-box: defrag or not;
which does not work retroactively.


>>> Similarly, perhaps we should note the value of available, and simply
>>> read that many bytes from the stream, regardless of how many bytes
>>> become available later.
>>>
>>> i.e.
>>>
>>> while ((got = in.read(readBuffer)) > -1 && !done) {
>>>     baos.write(readBuffer,0,got);
>>>     int a = in.available();
>>>     while (a > 0) {
>>>         got = in.read(readBuffer, 0, Math.min(readBuffer.length,a));
>>>         baos.write(readBuffer,0,got);
>>>         a = a - got;
>>>     }
>>>     interceptor.received(this, baos.toByteArray());
>>>     baos.reset();
>>> }
>>>
>>> What do you think?
>> I'm not sure what you meant by this.. ?
> Basically, the idea is that you note up front how many bytes were in the
> original packet (in.available()), read exactly that many, and pass those
> to the packet interceptor code.
>
> Rather than checking in.available() continuously, as that may
> theoretically change if another packet arrives while you are busy
> processing the first one.
>
> Of course, if you are unable to process packets as they arrive, and e.g.
> two are waiting the next time you come around to read bytes from the
> socket, in.available() will reflect the bytes for both pending packets,
> and they would automatically get coalesced, and there would be no way to
> know it. I guess that just means: read the packets as fast as you can,
> and hand off the processing of the packets to a different thread as
> quickly as possible.
Well, as I said, I don't see that much value in chasing exact packet
mapping. I don't really see the point, I can't really think of when I
myself would ever have needed it. But sure, there may be some value in
that in some cases (but then the destination better be just one hop
away, otherwise you can't be guaranteed that the packet will traverse
the network without fragmentation/defragmentation anyway (?))

Regards,
Martin


More information about the Owasp-proxy-project mailing list