[Esapi-user] ESAPI Random Number Generation Broken

Kevin W. Wall kevin.w.wall at gmail.com
Mon Apr 7 04:14:16 UTC 2014

On Sun, Apr 6, 2014 at 11:23 PM, Jim Manico <jim.manico at owasp.org> wrote:
> The problem is not thread safety, the original report had basic code that
> just returned 20,000 values from the ESAPI Randomizer API using the default
> configuration (SHA1PRNG) which came out non-random according to Burp's
> analyzer at the time.
> Several of my clients at the time acknowledged the same problem with
> ESAPI-Java and also used burp for analysis. We "fixed" it by just using new
> SecureRandom instances for CSRF tokens and other random number needs. I
> believe most of the ESAPI crypto API is using it's own internal SecureRandom
> instances.

I looked at this problem extensively back in 1Q2012. I was planning to reply
based on some tests that I ran using SecureRandom and Burp. Unfortunately,
that previous Christmas was when I gave my old laptop to my son and it was where
I was doing the testing (under a OpenSUSE installation) and I forgot about it
and when I finally went back to it, I discovered to my horrors that I
had stupidly
blown that partition away. I had Windows Vista, plus Ubuntu, Fedora,
and openSUSE
Linux installed. I had meant to blow away Ubuntu away [because I didn't like the
new Unity interface], but accidentally picked openSUSE. I had backups, but not
recent enough; they were all about a month old. So, anyway, I never reported
back on this, although there's been a email in my Draft folder for
more than 2 yrs
on this. Such is life; sigh.

Anyhow, I recall the gist of the test. I wrote a small program that
simply called
SecureRandom.nextBytes() [fetching 20 bytes at a time IIRC], base64 encoded
that result, and output it to stdout. I I repeated that in a loop with the same
SecureRandom instance, about 10,000 times. I then pumped the stdout into
Burp and it came up with something like 159-bits or so of entropy, or something
fairly close to the expected 160-bits. (I was using SHA1PRNG, the same
as ESAPI.)
That simple experiment told me
two things...1) that Burp was not *completely* insane, and 2) the problem was
NOT using the same instance of SecureRandom 10k times. (Trust me, if that were
the case, we'd have much huger problems not just with SecureRandom, but with
using SHA1 as a PRNG. I've read the 1.6 SecureRandom code...at least the
non-obfuscated parts and it seemed correct to me.)

I know there's been a lot of reference to Cigital's paper that was referred to
in some of these earlier threads, but that is misinterpreting the
paper. The reason
for periodic reseeding is not so that the output gains more entropy.
The reason is
if you use pretty much an CSRNG for too long an attacker _may_ be able to
use that to determine your initial seed and if they have that, then
they can predict
all future outputs. However, if the CSPRNG is actually thought to be secure (and
here we are talking about SHA1PRNG, which while having some collision issues,
should be okay for this purpose), if you use it for more than
2^ceil(L/2) with the same
seed and an adversary has seen all (or most) of those random #s you've
then you have a problem. For SHA1, L is 160, so 2^80 attempts is the absolute
max you would every want to use it and just to be safe, something much
more conservative,
like maybe 2^65 or so would be a safer bet.  Here, we are talking about 10^4 or
about 2^14 thereabouts. So that should definitely not be an issue at this #.

So IMO, because of these things, I think the whole using a SecureRandom
singleton instance is a red herring.

> Perhaps the problem is Burp's random analyzer?

Well, I think there's probably a bit of truth to that, or perhaps we
are not correctly
interpreting it's output. David Rooks original finding was that ESAPI's
DefaultRandomizer.getRandomString() was producing ZERO bits of entropy.
I, for one find that extremely hard to believe. That would pretty much
only happen
if all 10k (or 20k or however many samples that he was using) was 100%
E.g., if the same token was used each time. IIRC, he also said that it
didn't matter if
a 32-byte or 64-byte string was produced. Now, if he had found that
there were only
16bytes (128 bits) of entropy for the ESAPI results, I might be able
to believe it,
but IMO, the '0 bits' that Burp was showing indicated that either we
are misinterpreting
Burp's output or that the version we were testing with had a serious problem in
some of it's statistical tests. (For example, suppose it runs a
battery of 20 independent
tests and one of those tests failed for all input; e.g., maybe there
was several bits
correlated in some way. If that one test failed all trials, maybe it
causes it to report
'0 bits' of entropy even thought he other 19 statistical passed. E.g.,
if that one test that failed
only (say) detected some correlation in all the EVEN bit field #s
[bits 0, 2, 4, etc.],
that still might mean the odd bit fields are completely uncorrelated
and thus entropy
should be 1/2 of what it should be, but might be reported as 0 bits instead.)

Anyhow, I am sure that we can get to the bottom of this. My theory has
always been
that it is likely something in the ESAPI code that we are doing,
probably in transforming
it to a alphanumeric string consisting of A-Z, a-z, and 0-9. If it is
not that and turns out
to be something in the way we are using SecureRandom, I'd put my money
on the fact
that there is some bias in the base SecureRandom is converting it's
bytes to an int.
(E.g., maybe there's a bias in handling the sign bit, etc.)

Jim and I have resolved to work on this in the next week or so, but if
someone else
wants to help, pitch in. But I would encourage actual experimentation over

Thanks all for listening,
Blog: http://off-the-wall-security.blogspot.com/
NSA: All your crypto bit are belong to us.

More information about the Esapi-user mailing list