[Webappsec] Automatically detecting injections

[James Landis]

I agree that what we need to "prove" is definitely different depending
on the context, to answer your second question first. I would like to
address a point release and say: "SQLi is not possible for this
codebase." Others might be satisfied saying "here's my third-party
assessment saying I did my due diligence with respect to SQLi". Still
others might say "we don't care about SQLi because all of our users
are DBAs, anyway".

Whether or not what I want is possible...of course I agree
theoretically that you can't prove a negative. However, I believe that
it is possible to enumerate a set of 'safe' design patterns and flag
the rest for review. With each new release, you only end up reviewing
new lines of code for SQLi that are outside the set of safe patterns.
These then get whitelisted and ignored for future reviews. The
candidate list should become a vanishingly small subset after initial
security reviews, and might even disappear with a rigorous engineering
program.

I should say that 'safe' means 'not vulnerable to known classes of
SQLi', which is different from 'known SQLi signatures like OR 1=1'.
'Safe' means something like: only parameterized query procs from
vetted DB APIs are used, only with specific statement types, only with
canonicalized inputs. It means something specific and measurable, and
I believe that it is fully enumerable given what we know currently
about SQLi. Is anything really that new since 96 on the SQLi front?

On Mon, Apr 28, 2008 at 4:25 PM, Arian J. Evans
<arian.evans at anachronic.com> wrote:
> I'm not sure what "provable security" means, James.
>
>
> One degree of assurance is "free from defects", which cannot be proven.
>
>
> Another degree is "not what the hackers are currently exploiting".
>
>
> What we need to "prove" and to what degree seems to be the question.
>
>
> You cannot "prove" a negative. You are never going to
> be able to "prove" you are not vulnerable to SQL Injection.
> This is why we perform blackbox (fault-injection) testing.
>
> The best whitebox analysis can "prove" is to demonstrate
> you adhered rigorously to some model or implementation
> practice (if you did). Blackbox looks for omissions to this
> and possible emergent behaviors.
>
>
> As for the rest -- I didn't dismiss Jeff's sentiments. In fact,
>  as I stated: I agree with him in some cases.
>
>
> As for categorization, the traditional software testing definitions are:
>
>
> + Whitebox testing finds errors of commission
> + Blackbox testing tends to find errors of omission
>
> I was mixing the technical approach with the philosophy
> of analysis type, which aren't necessarily the same.
>
> --
>  --
>
> Arian J. Evans.
>
> I spend most of my money on motorcycles, mistresses, and martinis. The rest
> of it I squander.
>
>
>
>
> On Mon, Apr 28, 2008 at 3:29 PM, James Landis <jcl24 at cornell.edu> wrote:
>
> > In the event that you're trying to demonstrate "provable security"
> > with respect to SQLi, the only way to do that is using a tool or
> > process that directly operates on source code. You cannot prove that
> > an application is immune to SQLi using a runtime-only process. In the
> > event that you are just trying to demonstrate that an application does
> > not exhibit obvious SQLi signatures, runtime can be very good for that
> > in the sense that you can do it very quickly and very cheaply and
> > virtually irrespective of the codebase.
> >
> > I think it's important to know about possible agenda or bias, but also
> > not to entirely dismiss an argument just because of the potential bias
> > of the source. I agree that Jeff was somewhat oversimplifying the
> > argument when he said that it's possible to prove a lack of SQLi in a
> > matter of minutes with static analysis techniques. Unless he's got a
> > tool I haven't heard of, you can't be sure a parameterized query isn't
> > mapped to a sproc on the DB which is doing string concatenation on the
> > inputs, you can't be sure someone isn't passing in arbitrary data to
> > an exec (e.g. silky's example), and you can't be sure the codebase
> > isn't using some obscure API or custom hack job to talk directly to
> > the DB that doesn't use typical method signatures of mainstream APIs
> > that are already flagged as sinks for SQL queries. That is, for an
> > arbitrary application. However, if you're an engineering manager and
> > product security IS a concern, eventually you're going to get smart
> > and design the code in such a way that you CAN prove in a matter of
> > minutes with a source code tool that the application doesn't have a
> > SQLi vuln. I'd say the same is true for many other classes of
> > output-encoding-related issues. Maybe you wouldn't be able to prove it
> > with existing source scanners, but those aren't designed for that
> > right now, anyway.
> >
> > This is somewhat orthogonal to the discussion, but I think it muddies
> > the waters to think of "black box vs. white box" as interchangeable
> > with "runtime vs. static analysis". You can have both black-box and
> > white-box runtime-only assessments. You can have "no-box" static
> > analysis assessments, where you never touch running code to verify
> > findings. Certainly I would call a run-time + static assessment as
> > "white box", referring to the entire assessment. Does anyone else
> > think that this distinction is important or am I just splitting hairs?
> >
> > -j
> >
> >
> >
> >
> > On Mon, Apr 28, 2008 at 2:31 PM, Arian J. Evans
> > <arian.evans at anachronic.com> wrote:
> > > NNP: I recommend looking at the many SQL
> > >  Injection exploitation tools like SQL Power Injector:
> > >  http://www.sqlpowerinjector.com
> > >
> > >  The author is easy to reach, and on the WASC list,
> > >  if you have specific questions.
> > >
> > >  Secondly, for "blind" sql injection devoid of error
> > >  messages, but in which you can detect change
> > >  in state in some form (e.g. dynamic query on int
> > >  returns different news articles), you will need to
> > >  maintan some form of state and chain requests
> > >  to infer if your modified queries are running.
> > >
> > >  For "blind" SQL injection that doesn't provide any
> > >  obvious state changes, there are other approaches
> > >  -- the success of which wild depend upon your
> > >  privilege level, and the DB in question:
> > >
> > >  * sleep or wait a function
> > >  * have DB phone home
> > >  * check exploit tools above for many more ideas
> > >
> > >  ***
> > >  As Chair of OWASP, Jeff's voice carries weight
> > >  and his negative comments on blackbox testing
> > >  deserve some needed response.
> > >
> > >  Possibly Jeff's opinion is a result of the low quality
> > >  results and limited use-case most of the desktop
> > >  "scanning tools" (black box or source-code) provide.
> > >
> > >  I'd like to separate the low quality tools, and the
> > >  terrible consultant "assessments" we've all seen,
> > >  from the actual act of blackbox testing.
> > >
> > >  <comments=inline>
> > >
> > >
> > >  On Sun, Apr 27, 2008 at 3:08 PM, Jeff Williams
> <jeff.williams at owasp.org> wrote:
> > >  ]>  You can easily verify that an application with hundreds of
> thousands of
> > >
> > > >  lines of code doesn't have SQL injection in a matter of minutes. And
> you'd
> > >  >  be accurate. You just check to make sure it only uses a
> parameterized
> > >  >  database interface, like PreparedStatement.
> > >
> > >  >From my experience -- accurate analysis is not always this simple.
> > >  I have found high Severity exploitable SQL Injection in flagship
> > >  commercial products assessed by flagship consultancies due to
> > >  the fact they only verified use of bind variables or parameterized SQL.
> > >
> > >  In new frameworks and ideal design situations, it might be simple
> > >  enough to review for design patterns, but I have still found creative
> > >  implementation-level (exploitable) queries in software with otherwise
> > >  sound design/pattern specs or using an abstracted data-access layer.
> > >
> > >  Common cases I've seen include use of EXEC or sp_exec
> > >  on parameterized/bind-variable SQL, or any case where
> > >  parameters are SET to a string, or used as a passthrough
> > >  query to something that treats them as a concatenated string.
> > >  Maybe all possible cases of this are finite and easy to look for.
> > >
> > >  In general -- I tend agree with your *sentiment*. It is easier to find
> > >  and verify some types of SQL Injection issues via whitebox
> > >  review than blackbox testing. However, increase scope,
> > >  number of applications, and include apps with regular
> > >  updates to source code and I might flip my statement around.
> > >
> > >
> > >
> > >  >  As you start thinking up attack variants and encodings, you'll see a
> > >  >  combinatoric explosion in the fuzzer. Scans that used to take a few
> minutes
> > >  >  for a short list will start taking hours or days.
> > >
> > >  In general, again, I agree. Defect detection via stress testing
> > >  and fault injection -- which is what black box testing of software
> > >  is to manufacturing testing of materials and assemblies
> > >  -- this is always combinatoric. In terms of testing assemblies
> > >  (== combined running software modules) this is also a *strength*,
> > >  pragmatically, speaking, as you test "the sum of the parts".
> > >
> > >  Black box testing is also strong at finding emergent behaviors,
> > >  but that is for another discussion.
> > >
> > >
> > >
> > >  > And they're wildly  inaccurate.
> > >
> > >  Not necessarily. I do agree with you that the desktop scanners,
> > >  historically, are wildly inaccurate on SQL Injection.
> > >
> > >  It is possibly to black-box test, fuzz, and automate SQL Injection
> > >  testing to a very high degree of accuracy anywhere you can
> > >  measure state changes (explicit errors or not) in an application.
> > >  The tests I work with right now are in upper 90% percentile
> > >  of accuracy for SQL Injection.
> > >
> > >  Of course, if we argue false negatives for completely "blind"
> > >  SQL -- I have no idea what the real numbers are on this.
> > >  Nobody has any real numbers on pulse/frequency of occurance
> > >  of vulnerability per [$unit] to completely blind SQLi,
> > >  which makes this a difficult discussion.
> > >
> > >
> > >
> > >  >  This is true of many classes of security holes, but not all of them.
> XSS is
> > >  >  something that the tools have better luck with because it has to
> show up in
> > >  >  the response and it's relatively simple. But they still have trouble
> with
> > >  >  XSS that is stored in one place and retrieved in another, and
> complex XSS
> > >  >  that requires special tricks or encoding.
> > >
> > >  The fault here is in the *tools* and their poorly written *tests*,
> > >  not with black-box testing par se.
> > >
> > >  In terms of real-world production environments, with different
> > >  pieces of code, app servers, environmental conditions, and design
> > >  philosophies, all glued together (e.g.- also including intermediary
> > >  caching proxies, secondary data sources, etc. etc.) black box
> > >  analysis is essential.
> > >
> > >  I have been in the blackbox clean-up assessment slot many times
> > >  behind source code scanners or consultants and always find vulns.
> > >
> > >
> > >
> > >  >  I recommend you choose the most cost-effective approach (speed,
> accuracy,
> > >  >  coverage) for verifying each security area. For SQL injection, it's
> hard to
> > >  >  beat looking at the code.
> > >
> > >  Depending on the context -- I mostly agree. If someone were to
> > >  ask me:
> > >
> > >  "Help me find and patch as much of my sql injection as possible.
> > >  What should I do?"
> > >
> > >  My answer would entirely depend on their organizational
> > >  dynamics, logistics of source access, SDLC, and number
> > >  of software assets to assess ongoing.
> > >
> > >  If they have 50 or 100 applications, in production environments
> > >  spread around a data-center (let alone the globe) -- I would
> > >  say that source-code review is going to have as many, if not
> > >  more, challenges than black-box testing. Technical,
> > >  organizational, and logistic challenges to get and measure
> > >  the code and ensure it is production-like, to begin with.
> > >
> > >
> > >  > this is  In fact, it's pretty hard to beat code review for
> > >
> > > >  most problems. You'll hear lots of tool vendors claiming that code
> review is
> > >  >  so expensive.
> > >
> > >  Blackbox and Whitebox testing both have pros and cons,
> > >  and there is no clear 90/10 or 80/20 superiority rule that
> > >  fits all assessment cases. The cost of each entirely depends
> > >  on who you talk to, and how you measure your ROI.
> > >
> > >  It is beyond the scope of this thread to list all conditions
> > >  or caveats to where each approach shines...unfortunately
> > >  we do not have a guide called: "Methods of Software
> > >  Security Defect Analysis & How to Select Them"
> > >
> > >  In the manufacturing of complex machines, which I
> > >  believe is very akin to building software, you do *both*
> > >  whitebox and blackbox review. They are both essential.
> > >
> > >  You evaluate everything from design blueprints, to
> > >  materials specifications, to measuring and testing
> > >  parts, static assemblies, and running assemblies.
> > >
> > >  You do not just pick one method of analysis and
> > >  call that good for all occasions.
> > >
> > >
> > >
> > >  > Keep in mind that it's possible they have an agenda.
> > >
> > >  There are a lot of agendas right now on the web
> > >  application security lists, and none of them are
> > >  very clear to the layman.
> > >
> > >  OWASP, for example, is entirely dominated by
> > >  consultancies, tool vendors, and individuals that
> > >  perform source code review for a living, and that
> > >  bias shows strongly in much of the dialogue and
> > >  in the list items on the OWASP Top 10.
> > >
> > >  I'm not saying this is a bad thing -- I used to be
> > >  part of it, but it is not clearly evident to the
> > >  layman or person new to the industry.
> > >
> > >  Likewise, much of WASC is focused on attack
> > >  nodes and is blackbox centric (WAF/Scanner/etc)
> > >  and the focus of the members reflects this.
> > >
> > >  You also have the perspectives of academic
> > >  folks and the government standards people,
> > >  with ideas different from those enabling
> > >  business software, suggesting priorities
> > >  around what we should measure and how
> > >  we should address the issues.
> > >
> > >  There are many opinions on this subject
> > >  but very little math to back up or verify
> > >  any of our opinions beyond anecdotal
> > >  experiences we refer to (myself included)
> > >  if we are all being honest.
> > >
> > >  I have no fear of fuzzing myself. I've used
> > >  it for a long time now and it's served
> > >  me well on a number of occasions to
> > >  find complex, exploitable vulnerabilities.
> > >
> > >  --
> > >  Arian J. Evans, Software Security Stuff
> > >  --
> > >
> > >  Disclaimer of Agenda & Bias for Those Who Care:
> > >
> > >  I work on a blackbox testing framework and
> > >  service delivery for blackbox testing service
> > >  for roughly 600 web applications/year.
> > >
> > >  I've also performed source code reviews,
> > >  reverse engineering, binary analysis, and
> > >  whilst a terrible coder written everything
> > >  from XHTML to Rexx to Delphi to C# in
> > >  a variety of text editors on a variety of platforms.
> > >
> > >  I've also researched, published, and spoken
> > >  about vulns in commercial products including
> > >  network security devices, which is arguably
> > >  A LOT more fun than anything to do with Pascal.
> > >
> > >  cheers
> > >
> > >
> >
> >
> >
> >
> >
>