[Owasp-o2-platform] Why doesn't SAST have better Framework support (for example Spring MVC)?

Romain Gaucher rgaucher at cigital.com
Wed Oct 26 08:14:52 EDT 2011


Thanks to the ones asking me to send this email again. Hopefully that will
be readable now...


Everyone,

I originally asked this question to Dinis last Friday, and on his request,
I'm posting my reply to his last email.

On Sat, Oct 22, 2011 at 9:21 PM, dinis cruz <dinis.cruz at owasp.org>
<mailto:dinis.cruz at owasp.org>> wrote:

>There are a number of reasons why the tool vendors have not been able to
>provide decent (or even any) wide Framework Support on their tools
>
>Note that this is not for lack of trying, for example the latest version
>of AppScan already supports WAFL (Web Application Flow Language) which is
>their attempt  at creating a Framework descriptor language, HP is doing
>interesting work in their integration of WebInspect+Fortify and there are
>a couple new players (like WhiteHat, Veracode, Armorize) that claim will
>do a better job.

Right. I'm aware of Fortify to try to implement a framework support in
their SCA, though I could only realize the failure (i.e., limited
usability) of what they implemented. I did not know that AppScan had
something like this as well. WH/Veracode are out of my scope (I did not
have time to properly investigate their capabilities, and I assume they
did not have as much as time as the more mature players to address the
framework issues).

>For me, the key problem that all tools have (not only SAST, but this is
>critical in SAST) is that they are all trying to find a 'big red button'
>while ignoring how the app actually works/behaves. They basically want to
>create a product that can just be pointed to an application and work.

I totally agree with you. Even though it would be fantastic to have a
click-n-scan tool, with correct support for *anything*, it is clear that
they are trying to achieve orthogonal goals:
- Have a good support for languages
- Have an okay support for frameworks
- Have a good security weaknesses coverage

This definitely highlights a lack of *focus* to me, which is necessary in
good tools.

>
>The problem with this approach is that all apps are massively different!

Absolutely, I believe that Fortify/AppScan tried to implement a framework
description language that would be suitable for Struts, Spring, .NET,
Django, SproutCore, whatever. Though we know that those frameworks have
very different runtime behavior, even if their source-code presence is
somewhat similar.

>The apps themselves are build on top of MASSIVE frameworks (from a point
>of view of their behaviour), and even when they use common frameworks (vs
>writing their own frameworks), the way the actual code flows tends to be
>quite unique per app.
>
>So by trying to treat the "Application Behaviour' as a black box, and
>choosing NOT to try to (really) understand/map how it works (beyond the
>default Java/.NET functionality or whatever 'Framework Support' they are
>able to have), these tools are trying to climb a mountain that is far too
>big and complex.
>
>My approach with O2 has been "I know I will have to map how the
>application works/behaves and that I will need to create (from the
>source-code or dynamic analysis) an working model of its real
>code/data-flows, and while I'm there, also create a set of rules for the
>tools that I can use. My only question is: how long will it take to gain
>the level of visibility that I will need in order to be able to do a good
>job". This is what I call 'playing the Application Visibility game'

For my perspective, i.e. from a consultant point of view, your approach is
definitely better, and I've been trying to apply this model as well with
BlackSheep (dunno if you remember when I showed it to you), but only from
a black-box point of view.

>Basically with O2 I'm climbing a complete different mountain.
>
>Lets take for example Spring MVC. The first things I do when looking at a
>Spring app are:
>
> *   review the source code in order to 'codify' how the controllers are
>configured and what is their behaviour (namely the URLs, Command Classes
>and Views).
>    *   paying special attention to any 'Framework behaviour
>modifications', for example filters, authentication/authorization
>engines,  or even direct spring MVC code patches
> *   then I continue these mappings into the inner-working of the
>application in order to identify its 'hyper jumps' (reflection, aop,
>setters/getters, hash-objects-used-to-carry-data, web services,
>data/storage layers, other abstraction layers, etc...) and  'data
>changing' steps like validation or object casting.
> *   then I map out the connection between the controllers and the views
>(which is very important because we can't assume that there will be path
>into all views from all controllers)
> *   then....  (next actions depend on how the app is designed and what
>other APIs or Frameworks are used)
>
>When I'm doing these steps, I (using O2) tend to do three things:
>
> *   Create mini tools that visualize what is going on (for example url
>mappings to controllers, or the complete command classes
>objects<http://o2platform.wordpress.com/category/java/spring-mvc/> )
> *   Create Browser-Automation APIs that represent the expected behaviour
>of the target application (how to login, how to perform action XYZ, how
>to invoke a Web Service, etc...)
> *   Mass create rules for the tools available (for example I used to
>create 1000s of Ounce rules so that I would get the most of its Engine by
>getting it to create as many data-flow traces as possible
>
>So yes, I'm coding all the time
>
>The only difference between engagements, is that I'm able to build on the
>technology developed on the previous engagements.

>From what I understand here, you defined application specifics all the
time (this assumption is mostly derived from the browser-automation). The
danger to me here, is the non-reusability of the entire process. For
instance, why aren't you able, after all these iteration to combine all of
the steps into one process? I understand that the crawling might be tricky
and apps specific, but I truly (when helped with access to the source
code), that you can do a fantastic job at gathering all the entry points,
and triggering a proper functionality coverage.

>As you can see, although there is always some level of customization, its
>amount (and skill level) is reduced on each interaction (and this is how
>we will scale this type of analysis).

This is very related to my previous answer. In your opinion, what do you
need to do a better job at it? In order to scale faster, to have a better
coverage of the applications, etc.

>And here you can see why the SAST tools really struggle with frameworks,
>because they don't want to play this game. Ironically the end result is
>the same 'big button to press and get solid results' , the only
>difference is how to get there.
>
>My personal view (backed by real world experience) is that this is the
>only way that 'good enough' framework support can be added to a SAST tool
>in a way that it will actually be usable by developers.

Even though I agree with your first statement, I disagree with the second
paragraph. To me, we should just not throw a bunch of security weaknesses
(i.e., with 80% of FP) to developers, and this is the main problem. The
tools are trying to have a good coverage of security weaknesses, as well
as of frameworks.

The adoption would be much better if they focused on a very small area of
security issues, let's say XSS, SQLi, and few others, and trying to nail
those down properly for different frameworks. Over the years, there would
be a solid engine, with less noise.

That's really what I see is the missing point from most of the SAST
vendors (e.g., not Grammatech, Coverity, but the Fortify and such).
Developers are annoyed by the tons of findings, with stupid remediation
advices, and for which they need to waste their time to remove FP over and
over again.

>The good news is that I have shown with O2<http://o2platform.com/> how my
>proposed model can work in the real-work. It was done on top of an Open
>Source platform (O2), and it is out there for others to learn and copy
>
>Unfortunately, I am one of the few O2 users that can really do this, so
>the next step is to find a way to scale O2's techniques/usability and
>help SAST (and others) tools to develop/expose similar technology and
>wokflows.

Even though I like O2, the platform has a several limitations(in my
opinion), that refrain security consultants, and developers in using it. I
know this is a different discussion, but I believe that O2 could be really
used if it were to have:
- A workflow driven approach to security investigation/queries; for
example, if you had pre-given steps, where O2 could compute some paths,
then ask the developer to answer few questions, then continue the
analysis, and make security weaknesses decision;
- A less confusing UI (well, you might have improved it by now, but last
time I looked at it, last year, it was still pretty confusing);
- Focus the tool on static analysis or dynamic analysis (realistically,
few combined the two techniques) by implementing perspectives, or
something similar.

I know that's a ton of work for a one-man tool, but there must be a way to
involve the community. I'm definitely not an expert in OWASP/community,
but I know that what you see is like that all the time. Unless you have a
strong driver, and several supporters, it's difficult to get traction and
collaboration on a tool.

>Just imagine that we were able to use SAST tools in a way that they were
>really able to map/visualize/analyze an entire code/data flow, and create
>'solid, defensible and comprehensive' results (with very low False
>Positives and False Negatives)
>
>Don't you think the developers (and managers architects, buyers, consumer
>groups, government agencies,  etc..) would be ALL over it?

Well, who wouldn't be all over it. You describe a perfect tool for
consultants. IMO, developers, architects, managers, etc. need a very
different view of their app.

For a developer, it needs the localized information: what he is currently
working on. As a matter of fact, dataflow information about what the
interaction is with the framework (stuff he doesn't control), the rest of
the team (his colleagues) would be quite important. But they must be
focused.

An architect, would need a component view of the app. He does not want to
dig into the code, but to see the relationship between components, what
security assumptions are made when communicating between pieces, etc.

Does this make sense to you?

Thanks,
Romain

--
Romain Gaucher
Sr. Consultant, Software Security
www.cigital.com



More information about the Owasp-o2-platform mailing list