[Owasp-o2-platform] Why doesn't SAST have better Framework support (for example Spring MVC)?
rgaucher at cigital.com
Tue Oct 25 16:02:43 EDT 2011
I originally asked this question to Dinis last Friday, and on his request, I'm posting my reply to his last email.
On Sat, Oct 22, 2011 at 9:21 PM, dinis cruz <dinis.cruz at owasp.org<mailto:dinis.cruz at owasp.org>> wrote:
There are a number of reasons why the tool vendors have not been able to provide decent (or even any) wide Framework Support on their tools
Note that this is not for lack of trying, for example the latest version of AppScan already supports WAFL (Web Application Flow Language) which is their attempt at creating a Framework descriptor language, HP is doing interesting work in their integration of WebInspect+Fortify and there are a couple new players (like WhiteHat, Veracode, Armorize) that claim will do a better job.
Right. I'm aware of Fortify to try to implement a framework support in their SCA, though I could only realize the failure (i.e., limited usability) of what they implemented. I did not know that AppScan had something like this as well. WH/Veracode are out of my scope (I did not have time to properly investigate their capabilities, and I assume they did not have as much as time as the more mature players to address the framework issues).
For me, the key problem that all tools have (not only SAST, but this is critical in SAST) is that they are all trying to find a 'big red button' while ignoring how the app actually works/behaves. They basically want to create a product that can just be pointed to an application and work.
I totally agree with you. Even though it would be fantastic to have a click-n-scan tool, with correct support for *anything*, it is clear that they are trying to achieve orthogonal goals:
- Have a good support for languages
- Have an okay support for frameworks
- Have a good security weaknesses coverage
This definitely highlights a lack of *focus* to me, which is necessary in good tools.
The problem with this approach is that all apps are massively different!
Absolutely, I believe that Fortify/AppScan tried to implement a framework description language that would be suitable for Struts, Spring, .NET, Django, SproutCore, whatever. Though we know that those frameworks have very different runtime behavior, even if their source-code presence is somewhat similar.
The apps themselves are build on top of MASSIVE frameworks (from a point of view of their behaviour), and even when they use common frameworks (vs writing their own frameworks), the way the actual code flows tends to be quite unique per app.
So by trying to treat the "Application Behaviour' as a black box, and choosing NOT to try to (really) understand/map how it works (beyond the default Java/.NET functionality or whatever 'Framework Support' they are able to have), these tools are trying to climb a mountain that is far too big and complex.
My approach with O2 has been "I know I will have to map how the application works/behaves and that I will need to create (from the source-code or dynamic analysis) an working model of its real code/data-flows, and while I'm there, also create a set of rules for the tools that I can use. My only question is: how long will it take to gain the level of visibility that I will need in order to be able to do a good job". This is what I call 'playing the Application Visibility game'
For my perspective, i.e. from a consultant point of view, your approach is definitely better, and I've been trying to apply this model as well with BlackSheep (dunno if you remember when I showed it to you), but only from a black-box point of view.
Basically with O2 I'm climbing a complete different mountain.
Lets take for example Spring MVC. The first things I do when looking at a Spring app are:
* review the source code in order to 'codify' how the controllers are configured and what is their behaviour (namely the URLs, Command Classes and Views).
* paying special attention to any 'Framework behaviour modifications', for example filters, authentication/authorization engines, or even direct spring MVC code patches
* then I continue these mappings into the inner-working of the application in order to identify its 'hyper jumps' (reflection, aop, setters/getters, hash-objects-used-to-carry-data, web services, data/storage layers, other abstraction layers, etc...) and 'data changing' steps like validation or object casting.
* then I map out the connection between the controllers and the views (which is very important because we can't assume that there will be path into all views from all controllers)
* then.... (next actions depend on how the app is designed and what other APIs or Frameworks are used)
When I'm doing these steps, I (using O2) tend to do three things:
* Create mini tools that visualize what is going on (for example url mappings to controllers, or the complete command classes objects<http://o2platform.wordpress.com/category/java/spring-mvc/> )
* Create Browser-Automation APIs that represent the expected behaviour of the target application (how to login, how to perform action XYZ, how to invoke a Web Service, etc...)
* Mass create rules for the tools available (for example I used to create 1000s of Ounce rules so that I would get the most of its Engine by getting it to create as many data-flow traces as possible
So yes, I'm coding all the time
The only difference between engagements, is that I'm able to build on the technology developed on the previous engagements.
>From what I understand here, you defined application specifics all the time (this assumption is mostly derived from the browser-automation). The danger to me here, is the non-reusability of the entire process. For instance, why aren't you able, after all these iteration to combine all of the steps into one process? I understand that the crawling might be tricky and apps specific, but I truly (when helped with access to the source code), that you can do a fantastic job at gathering all the entry points, and triggering a proper functionality coverage.
As you can see, although there is always some level of customization, its amount (and skill level) is reduced on each interaction (and this is how we will scale this type of analysis).
This is very related to my previous answer. In your opinion, what do you need to do a better job at it? In order to scale faster, to have a better coverage of the applications, etc.
And here you can see why the SAST tools really struggle with frameworks, because they don't want to play this game. Ironically the end result is the same 'big button to press and get solid results' , the only difference is how to get there.
My personal view (backed by real world experience) is that this is the only way that 'good enough' framework support can be added to a SAST tool in a way that it will actually be usable by developers.
Even though I agree with your first statement, I disagree with the second paragraph. To me, we should just not throw a bunch of security weaknesses (i.e., with 80% of FP) to developers, and this is the main problem. The tools are trying to have a good coverage of security weaknesses, as well as of frameworks.
The adoption would be much better if they focused on a very small area of security issues, let's say XSS, SQLi, and few others, and trying to nail those down properly for different frameworks. Over the years, there would be a solid engine, with less noise.
That's really what I see is the missing point from most of the SAST vendors (e.g., not Grammatech, Coverity, but the Fortify and such). Developers are annoyed by the tons of findings, with stupid remediation advices, and for which they need to waste their time to remove FP over and over again.
The good news is that I have shown with O2<http://o2platform.com/> how my proposed model can work in the real-work. It was done on top of an Open Source platform (O2), and it is out there for others to learn and copy
Unfortunately, I am one of the few O2 users that can really do this, so the next step is to find a way to scale O2's techniques/usability and help SAST (and others) tools to develop/expose similar technology and wokflows.
Even though I like O2, the platform has a several limitations(in my opinion), that refrain security consultants, and developers in using it. I know this is a different discussion, but I believe that O2 could be really used if it were to have:
- A workflow driven approach to security investigation/queries; for example, if you had pre-given steps, where O2 could compute some paths, then ask the developer to answer few questions, then continue the analysis, and make security weaknesses decision;
- A less confusing UI (well, you might have improved it by now, but last time I looked at it, last year, it was still pretty confusing);
- Focus the tool on static analysis or dynamic analysis (realistically, few combined the two techniques) by implementing perspectives, or something similar.
I know that's a ton of work for a one-man tool, but there must be a way to involve the community. I'm definitely not an expert in OWASP/community, but I know that what you see is like that all the time. Unless you have a strong driver, and several supporters, it's difficult to get traction and collaboration on a tool.
Just imagine that we were able to use SAST tools in a way that they were really able to map/visualize/analyze an entire code/data flow, and create 'solid, defensible and comprehensive' results (with very low False Positives and False Negatives)
Don't you think the developers (and managers architects, buyers, consumer groups, government agencies, etc..) would be ALL over it?
Well, who wouldn't be all over it. You describe a perfect tool for consultants. IMO, developers, architects, managers, etc. need a very different view of their app.
For a developer, it needs the localized information: what he is currently working on. As a matter of fact, dataflow information about what the interaction is with the framework (stuff he doesn't control), the rest of the team (his colleagues) would be quite important. But they must be focused.
An architect, would need a component view of the app. He does not want to dig into the code, but to see the relationship between components, what security assumptions are made when communicating between pieces, etc.
Does this make sense to you?
Sr. Consultant, Software Security
More information about the Owasp-o2-platform