[Esapi-user] [Esapi-dev] ESAPI 2.0 crypto documentation

Kevin W. Wall kevin.w.wall at gmail.com
Sat Apr 17 10:50:18 EDT 2010

Mike Boberski wrote:
> I've been putting some thought into ESAPI crypto lately, as I've been
> working on the PHP port...
> I think we've been not looking at crypto in as big-picture a way as the
> other controls have been looked at (I am certainly guilty of that, latching
> onto e.g. FIPS 140-related implementation details), and I think the
> below/something along those lines will further promote the use of ESAPI to
> wrap crypto.
> In other words, while the 2.0 crypto is technically excellent by anyone's
> measure, it is not ESAPI-ish (i.e., "make it as easy as possible") enough.

Well, I agree in part with what you are saying here. _Overall_ (speaking
more broadly than just the symmetric encryption methods) I would agree
that some things are not as "easy" as the rest of ESAPI, and probably
this is true of the functionality related to Encryptor in general.
(Neither is it as flexible as it should be, but that's a completely
different topic, so I will not digress here.)

To a large degree, this apparent complexity in the crypto related
functionality of ESAPI has to do with the general development community
not having a great deal of awareness about cryptographic primitives and
what they are useful for. A good example is the use of the method name
'hash' in Encryptor. Most developers are going to think of "hash" as
something that happens inside a Hashtable or HashMap rather than creating a
cryptographically-strong, secure, one way message digest. And yet, if
these same developers would read any of the cryptographic literature--even
that aimed at the generic development community such as you might
find in _Communications of the ACM_--you will find that those authors
use the same terminology. So with crypto, I think there's a larger than
normal knowledge gap to cross compared to the concepts in the rest of

> *Discussion item #1. *I think the following function names would be more
> ESAPI-ish:
>    - protectFromDisclosure
>    - protectFromDisclosureAndModification
>    - protectFromDisclosureWhileAtRest
>    - protectFromDisclosureAndModificationWhileAtRest
>    - protectFromDisclosureWhileInTransit
>    - protectFromDisclosureAndModificationWhileInTransit
>    -
>    - unprotectFromDisclosure
>    - unprotectFromDisclosureAndModification
>    - unprotectFromDisclosureWhileAtRest
>    - unprotectFromDisclosureAndModificationWhileAtRest
>    - unprotectFromDisclosureWhileInTransit
>    - unprotectFromDisclosureAndModificationWhileInTransit
>    - ... maybe then additional qualifier-ish names,
>    e.g. protectFromDisclosureInOracle (maybe output format is in some optimized
>    format for Oracle for example), protectFromDisclosureInPDF (maybe wrap
>    interesting things like this), etc. ...
>    - ... maybe then additional variations,
>    e.g. protectFromDisclosureUsingMaximumKeySize, etc. ...

I am open to having a discussion on this, however, I think you need to be
a bit more specific of what methods you are referring to.

Kindly expound upon the above suggested names if you would, say something
old_name --> new_name.  For example, I might, for the the purposes of this
discussion at least, presume that you are suggesting that the various
Encyptor.encrypt() methods be renamed to Encryptor.protectFromDisclosure() ???

If that's the case, I would think that the resulting signature like
    CipherText Encryptor.protectFromDisclosure(SecretKey, PlainText)

also would also need changed to rename the classes CipherText, SecretKey,
and PlainText to something else as well.  And at that point, while you
_might_ get something that hides the jargon from those with no prior
exposure to cryptography, you also _might_ risk loosing those who have
some reasonable fluency with the subject. (Certainly my first reaction
if I ran across an API with names like this might be to think of the
interfaces need to be simplified to this extent, then how do I know
that the "protecting from disclosure" isn't simply something based
on some home-grown obfuscation. I probably would be reluctant to use
an API with names like this if I did not almost immediately run across
some very clear documentation that ensured me that it was using proven
encryption and not just obfuscation under the hood.)

Back in 1999, I started developing what 2 years later became a proprietary
Java class library of something that was very much ESAPI-like.  My team has
been refining this class library and its documentation and supported examples
ever since 2001 or so. (We also support a similar .NET implementation with
corresponding class and method names.) For the first 5 years or so, this
class library was about 80% cryptographic related and 20% data validation
related. During that time, we had many discussions, not only amongst our
many team members (who all started out as developers with no prior security
or cryptography experience), but also with many dozens of developers from
the IT department at Qwest. Those discussions resulted in the following
key observations:
    1) Most IT developers preferred that we use names that other APIs such as
       the Microsoft Cryptography APIs, Suns Java Cryptography Extensions, etc.
       used rather than coming up with something different.
    2) Make the API defaults secure (e.g,. use AES by default), but allow
       developers to use the API with other various, perhaps less secure,
       cryptographic options (.e.g., DES) in case they need to inter-operate
       with legacy applications or third-party software.
    3) As much as possible, remove the inherent complexities of cryptography.
       For example, don't make developers learn about cipher modes, IVs,
       padding schemes, how to validate certificates, etc., unless this
       understanding is essential in accomplishing the desired functionality
       securely. (This is closely related to secure defaults.)
    4) Do not assume that you know the context of why or where developers are
       using your methods. For example, don't have one method to encrypt a
       string and another method to encrypt a URL. Don't have one method to
       encrypt something to store to a file and another to encrypt something
       to store in a database. Instead, the dev teams wanted us to give them
       the necessary primitives and they would wrap our APIs with appropriate
       wrappers of their own making.
    5) Provide dev teams with APIs that will allow them to do things portably.
       For example, give them the ability to encrypt anything on a big-endian
       SPARC V9 running Solaris and send that to a little-endian x86 running
       Windows where they can decrypt it and don't make them have to handle
       differences in endian-ness, or signed vs. unsigned. etc.

Of all these observations, the one that initially surprised me the most was
#1, followed by #4.

I think that in some regards, that observations #1 and #4 are related. First of
all, there is this tendency by the development community to think that they may
need to provide some related functionality that your API does not provide or
that at some point in the future, your API may no longer be supported. So there
this tendency to wrap your API (#4) whether it initially makes a lot of sense
or not. This application wrapping of your APIs not only allows them to
provide some application context (e.g., an encryptAndSecurelyStoreCreditCard()
method), but it also provides a layer of de-coupling between their code and
your API. By doing so, they can easily swap out your API for another one
at almost anytime. This only makes sense. It would not make much sense for
us to provide a bunch of "encryptAndSecurelyStoreX()" methods because not
only are there a lot of common 'X' (SSN, CC#, PII, etc.), but also where/how
these things are stored differs greatly from application to application.
Perhaps when we observe that we have a certain level of applications all
doing the exact same thing in this regard (perhaps because of some specific
compliance or regulatory issue or because of some InfoSec policy or business
requirement) we will offer to incorporate said functionality into our API,
but in general we try not to cross that line and know specifically how the
application will be using our building blocks. Usually we find it best to
leave the higher level abstractions to the application development team.

Secondly, I think that this at least in part explains observation #1. By
using the same terminology of other similar APIs, especially in the
case of cryptographic terminology, it makes it much easier for them
to resort to using another API such as MS-CAPI or Sun JCE or Peter Gutmann's
cryptlib, or openSSL, etc.

Since most of these other cryptographic APIs also use the terminology of
the cryptographic community, using these terms in our API serves to introduce
developers to these concepts so then if they run across an article in
something like _CACM_ or an IEEE CS proceedings on cryptography they will
have already been exposed to some of the fundamental terminology and they
will then find themselves on more familiar ground.  So, while using
domain specific jargon may not seem like the right thing to do in the
short run, it seems to generally pay off in the long haul.

Soooo... I am not saying that we should discard what you are suggesting, but
rather that we need to proceed cautiously.

Perhaps we need both "layers"...one that is more by intent, from a functionality
perspective, similar to what you suggest and the other using the more
traditional nomenclature like presently have. That's why I asked you for more
specific 'oldName --> newName' suggestions because it would help me to see that.

There certainly are places in ESAPI (and not just related to crypto) that
we need to think revising names. We've already had a lot of discussion on both
of these lists about this issue. I think that we need to encourage continued
discussion in this area. However, we need to also keep in mind that at some
point we need to come to closure and solidify what is in the official 2.0
release and push out other plans to either a later point release (2.1) or
later major release (3.0). That is needed because there are many companies
who are nervous (mine being one of them) of deploying "release candidate"
FOSS to a production environment. *BUT*, once we have something in an
official release, such as the 2.0 GA release, then we need to make a commitment
to any API changes and follow a standard deprecation policy so we aren't just
yanking the rug out from underneath our user base.  I know that you (Mike)
understand this, but am just mentioning it for those who have not been following
our discussions on deprecation policies, etc. for the past 4 to 6 months or so.

> *Discussion item #2. *I think we need to do better with key management;
> without going into gory detail or proposing something too hard to implement,
> I think crypto keys need to be associated with User objects as
> properties/attributes .
>    - Users (likely in many cases users representing server apps) will need
>    to be authenticated and logged in with all of the existing ESAPI plumbing
>    (logging, intrusion, etc.) before able to access keys
>    - This will make clear associations of keys, allow easily for multiple
>    keys (Encryptor would just use keys of whomever is logged in), and again
>    leverage all the existing plumbing.

I'm glad that you agree with this. I think that this is a good direction
to head in. In particular, I would prefer that my PII and other sensitive
information be encrypted with a key that is different than the one encrypting
your key. I've been arguing this point at Qwest since about 2001, but so far,
it's fallen mostly on deaf ears and has to do mostly with legacy code and
legacy reports.

But there will also be the need to have "per application" keys as well. Both are
needed. At my day job, we almost exclusively use the application keys and very
seldom use per user keys. But I do think that it's a good idea and can take
using cryptography to the next level.

Fortunately, now that we have cryptographic primitives that can take a SecretKey
object, we have the ability to build out these others. The original presumption
back in ESAPI 1.4 and carried initially into the early 2.0 release candidates
that there was only a single master key per application and everything to be
encrypted or decrypted had use that specific key was way too simplistic to
say the least.

The down side to all these additional keys is that you need a way to securely
manage them.

Ideally we need to allow ESAPI to be used with some sort of secure key
management system. I think this is doable, whether it is using something
like KeyCzar (http://www.keyczar.org/) or
StrongKey (http://sourceforge.net/projects/strongkey/) or your favorite
proprietary key management system or something yet to be determined, is,
well something TBD. But let me assure you that I have been thinking
about it. In fact, not only have I been thinking about it, but I have already
done something like this (and beyond) at my day job.  So it's on the radar,
but certainly will *NOT* be a part of 2.0.

> *Discussion item #2. *I think we need to standardize on PKCS#7 (ideally CMS)
> formats

Am assuming you meant to label this one 'item #3' since the previous discussion
was labeled '#2'.

>    - Then, include a parameter in e.g. protectFromDisclosure to return
>    binary or base 64.
>    - Perhaps further optimize with functions where this parameter would then
>    be hardcoded as mentioned above for the
>    example protectFromDisclosureInOracle

Yes, I would like to start using more standards such as CMS (RFC 3852) or

What makes a lot of that difficult is the ambitions to support ESAPI across so
many programming languages. Many of these support C or C++ interfaces where
there's lots of support for these things, but some perhaps not so much.
But if we had to implement things like CMS ourselves that can get into quite
a bit of scaffold building. And I would like to at least ensure that we have
some level of portability of supporting encryption / decryption across different
programming language versions of ESAPI. It would be presumptuous to assume that
any given IT department only uses on programming language to the exclusion of
all others.

As far as support for asymmetric cryptographic primitives goes (e.g.,
signing / validating digital signatures, public key encryption, etc.),
I'd like to see that functionality extended to be able to use
X.509 certificates from PKCS#12 key stores.  That is much more generic
than what we presently have and provides a higher degree of assurance
as well.

Anyway, good beginnings of a discussion. Keep it going.

Kevin W. Wall
"The most likely way for the world to be destroyed, most experts agree,
is by accident. That's where we come in; we're computer professionals.
We cause accidents."        -- Nathaniel Borenstein, co-creator of MIME

More information about the Esapi-user mailing list