[Owasp-modsecurity-core-rule-set] Announcing: Real-time Application Profiling Ruleset

Colin Watson colin.watson at owasp.org
Mon Mar 21 05:08:49 EDT 2011


Comments inline.  If we need to discuss any of these further (and most
don't), perhaps we should start separate threads?

On 18 March 2011 13:36, Ryan Barnett <RBarnett at trustwave.com> wrote:
> On 3/16/11 5:21 AM, "Colin Watson" <colin.watson at owasp.org> wrote:
>>I have implemented the experimental rules on a couple of servers, one
>>in a production environment.
> Does it seem to be working as expected?

I should have said so - yes it works as expected.

>>Having read the blog post at:
>>and your reply to Lucas in this thread, I have some additional
>>comments.  None of these are meant to criticise the excellent work so
>>far, and may well reflect ideas the team already has, or are just
>>restating concepts in related work from Ivan Ristic, Ofer Shezaf and
>>Christian Bockermann.  Some of these are of course just a dream wish
>>list.  Anyway here goes...
> Feedback is always welcome!  This version of the rules was merely the
> first step.  There are many improvements that we can make.
>>C1: I had to create an empty "modsecurity_40_profiler_ignore.data" for
>>the rules to work "as is" i.e. this file is not in the CRS 2.1.2
> Ah right.  I will update the comment text for "Ignore Resource" in the
> file that if the file is not present that you have to create one.  I guess
> we could have the default also be that these two SecRules are commented
> out by default.  If a user wants to ignore profiling/enforcement for
> certain resources, then they can uncomment these lines and then create the
> .data file.
>>Is the syntax of this file one URL per line?
> Yes - see the Notes section of @pmFromFile -
> http://sourceforge.net/apps/mediawiki/mod-security/index.php?title=Referenc
> e_Manual#pmFromFile

Great thanks, I should have looked there.

>>C2: Apart from ignoring certain URLs, could we specify some parameters
>>to ignore (on some URLs) and on some HTTP methods?  i.e. ignore params
>>if GET, but enforce if POST.
> In ModSecurity v2.6 (available in trunk) we added the
> SecRuleUpdateTargetById (and ctl:ruleUpdateTargetById) -
> http://sourceforge.net/apps/mediawiki/mod-security/index.php?title=Referenc
> e_Manual#SecRuleUpdateTargetById
> This capability will allow you to create chained rules with the
> ctl:ruleUpdateTargetById to dynamically exclude ARGS from validation.
> Of course in looking at these rules, we do need to assign rule IDs to them
> for this to work.

Noted, thanks.

>>C3: Could the thresholds (50 and 100) be set near the top of the rule
>>file, instead of within the SecAction of Step 1?
> Well, it is the first configurable setting in the file.  What issue would
> this solve?  We want to try and keep consistency with the CRS banner info
> at the top and then I wanted to have a small intro section of text about
> the detection.

You're probably right... they are not that difficult to find/set.

>>C4: Thresholds per host/site?
>>C5: Some application owners may know all the allowable entry
>>points.... it would be fantastic to have a way of defining that
>>profile (in XML?) which the CRS profiler could
>>   i) use as a base starting point, or
>>   ii) use as a mandatory profile, or
>>   iii) use to compare against when reporting discrepancies.
>>  This would also help to group similar URLs (e.g. paths when URL
>>rewriting is in use).
> So if an organization actually had an XML map/profile of their entry
> points right?  Hmm, you know what would be cool is to have something
> similar to @validateSchema but called something like @validateProfile.
> This would be able to parse an XML file that defines the allowable site
> tree profile and then block/alert when request data that doesn't match.
> That would be pretty cool.  :)

Yes it would.  Even on complex sites, it might be possible for source
code scanning tools to identify all the entry points (and types), and
they might be able to export a profile.

I currently export sets of ModSecurity rules to do this for one site
where the entry points are defined in a database.  But it's not very
efficient, and a profile schema/processor would be better.

The specified profile could also be used to compare against the
learned profile to look for gaps and discrepancies.

>I guess the next step would be to come to
> an agreed upon Site profile schema.  I think this is where your references
> to Ivan's Recipes (http://www.modsecurity.org/projects/ppr/) and Chris
> Bockermann's Web Profiles (http://jwall.org/web/policy/index.jsp) comes in
> as we would need a standard method of defining an app profile.

Yes, that would be the way forward - maybe a separate project?

>>C6: Ability to modify model in C5 in real time (e.g. as URLs are added
>>created when writing a new blog post, or pages are added in a CMS)
> Agreed.  The way it is implemented currently with the CRS profiler rules,
> however, is on a per-resource basis anyways.  So if you have a new
> resource that is dynamically created by a CMS then it will begin to be
> profiled.
>>C7: Ability for model in C5 to have the idea of "strict" and
>>"allowable" parameter value definitions.  Example 1: we might want a
>>phone number to be all numbers and spaces, but let's not get too upset
>>with hyphens and parentheses.  Example 2: A trailing line break on a
>>password could just be from copy & paste, not an attack.  So the
>>profiler would then need to report whether a parameter value just met
>>the allowable format, or failed that too.
> This is referencing the AppSensor model of categorizing the event as
> either Suspicious or Malicious.  As you can see in the rules, I have
> mapped each rule to the corresponding AppSensor Detection point.  For
> example -
> tag:'OWASP_AppSensor/RE1'
> My view is that these profiler rules only identify Suspicious events as
> they don't match an expected learned profile.  In order to determine
> Malicious events, the CRS should be run in Anomaly Scoring Mode
> (http://blog.spiderlabs.com/2010/11/advanced-topic-of-the-week-traditional-
> vs-anomaly-scoring-detection-modes.html) so that the transaction continues
> into the other blacklist CRS rules to verify if there are XSS, SQLi attack
> payloads, etc...  The profiler rules will increase an anomaly score and it
> is set at a slightly lower severity level than attack rules (that have a
> critical level) -
> setvar:tx.profiler_score=+%{tx.error_anomaly_score}

Of course if the profile was defined, rather than learned, we could be
more definitive about malicious vs. suspicious.  But we'd also have to
take into account non-malicious URL requests (e.g.
) giving a softer edge to the application's profile.

>>M1: Accessing the same URL on multiple virtual hosts, using the same
>>instance of ModSecurity (and with the same web browser) seems to
>>create an aggregated profile rather than a profile per host (but I may
>>be wrong):
>>  http://www.example1.com/profiler.txt?site=2
>>  http://www.example2com/profiler.txt?site=1&day=wednesday
>>  Is there a way to profile each host separately?
> Good point - I think that we need to expand the RESOURCE collection data
> so that it can utilize the SecWebAppId directive data in the same way that
> the setsid does so the collections are per-site.


>>M2: Could (or Is?) the HTTP method be part of the profile so there
>>could be different parameters for GET and POST for example.
> I see your point as there may be different parameters that are valid based
> on the Request Method.  We could do that with a new SecAction line at the
> beginning like this -
> SecAction
> "phase:1,t:none,nolog,pass,initcol:resource=%{REQUEST_FILENAME}_%{REQUEST_M
> ETHOD},setvar:resource.pattern_threshold=50,setvar:resource.confidence_coun
> ter_threshold=100"
> The only issue I see is if a client sends a request to a resource with a
> different REQUEST_METHOD, then it will create its own resource collection.
>  So how would we validate which request methods are valid/allowed?  I
> guess we could rely upon the app's response (HTTP Status Code of 405 -
> Method Not Allowed) to let us know if the request was valid in this case
> and then remove the collection.

Yes there are difficulties with this, and I don't have good answers
for profiling.

>>M3: Further data types (bit, SQL types, UUID, lists of integers, etc),
>>and perhaps user-definable data types - both of these would be of more
>>use if C5 is implemented.
> I have a long list of other parameter types to add but I wanted to this
> framework out for testing.

Yes, I guessed so!

>>M4: Ability to allow null values for parameters e.g. normally 0 or 1
>>but could be NULL
> Yeah, the RegEx length range checks are a but crude... Do you mean that a
> param payload could be:
> - foo.php?param=1
> - foo.php?param=0
> - foo.php?param

Yes, and

- foo.php?param=

in case that's different from your last example.

> In these cases, the parameter named "param" might match both of these
> length checks -
> SecRule ARGS "^0$"
> "phase:5,t:none,t:length,nolog,pass,setvar:resource.args_length_check0_%{ma
> tched_var_name}=+1"
> SecRule ARGS "^[1-5]$"
> "phase:5,t:none,t:length,nolog,pass,setvar:resource.args_length_check1_%{ma
> tched_var_name}=+1"
> And then added to either the resource.args_length_check0_enforce or
> resource.args_length_check1_enforce variable lists for enforcement.  It
> would depend upon how often the param was empty/NULL.

>>M5: Build result into anomaly scoring.
> You mean something more than what is already there?  We already have two
> levels of anomaly scoring built it -
> - Contribution to the overall anomaly score
> setvar:tx.anomaly_score=+%{tx.error_anomaly_score}
> - Contribution to a "profiler" anomaly score which would allow a user to
> add custom blocking level just for profiler check scores.
> setvar:tx.profiler_score=+%{tx.error_anomaly_score}

No, you are correct - this is sufficient as it is.

>>Data Storage
>>D1: Do you have any references for the format of this file, and of the
>>profiler key records, so we can try to extract data and convert it
>>into other formats?
> You mean the data stored in the RESOURCE collections?  If so, I would
> recommend using Christian Bockermann's jwall-tools java app so you can
> view collection data outside of the Debug log -
> https://www.jwall.org/tools/jwall-tools.jsp.  Perhaps Chris could update
> the tools to extract RESOURCE collection data and translate it to an XML
> profile.
> Another cool idea would be for the jwall-tools to be able to actually
> manipulate the RESOURCE collection data!  That would allow for some manual
> tweaking of a profile rather than having to just delete and relearn it.

That's sounds interesting, I'll take a look.

>>Syndication of result
>>S1: Could the "result" be added as another X-WAF-... Header
> They should already be added to the default list of match events.  But if
> you want a separate one, we could do that.

Thinking to the future, there may be more information we want to send
regarding the Profiler findings (actual parameter results, comparison
to learned profile, comparison to defined profile).  Therefore a
separate item now might be of use (as well)?

>>S2: This one isn't Profiling-specific, but currently the X-WAF-
>>headers are added to the request... could there be a method to pass
>>data on by URL (e.g. like CSP reporting mechanism), so that say even
>>if something is blocked it is possible to inform other devices of the
>>event?  The data sent could be configurable, but might include what's
>>currently in the X-WAF- headers, plus the original header content.
> Just so I understand - we would block the original request but then send a
> new HTTP request out from the WAF to other security devices with the X-WAF
> data as URL parameters?

Yes, but also even if ModSecurity doesn't block... potentially for all
requests.  The HTTP requests might be to a different host/port than
the application being protected.

>  You should be able to do that by using the
> "exec:" action and just write a script that will use something like
> curl/wget to send the HTTP request and populate the data you need.  I will
> test this out.

That sounds like the way.



More information about the Owasp-modsecurity-core-rule-set mailing list