[Owasp-modsecurity-core-rule-set] Announcing: Real-time Application Profiling Ruleset

Ryan Barnett RBarnett at trustwave.com
Fri Mar 18 09:36:48 EDT 2011

On 3/16/11 5:21 AM, "Colin Watson" <colin.watson at owasp.org> wrote:

>I have implemented the experimental rules on a couple of servers, one
>in a production environment.

Does it seem to be working as expected?

>Having read the blog post at:
>and your reply to Lucas in this thread, I have some additional
>comments.  None of these are meant to criticise the excellent work so
>far, and may well reflect ideas the team already has, or are just
>restating concepts in related work from Ivan Ristic, Ofer Shezaf and
>Christian Bockermann.  Some of these are of course just a dream wish
>list.  Anyway here goes...

Feedback is always welcome!  This version of the rules was merely the
first step.  There are many improvements that we can make.

>C1: I had to create an empty "modsecurity_40_profiler_ignore.data" for
>the rules to work "as is" i.e. this file is not in the CRS 2.1.2

Ah right.  I will update the comment text for "Ignore Resource" in the
file that if the file is not present that you have to create one.  I guess
we could have the default also be that these two SecRules are commented
out by default.  If a user wants to ignore profiling/enforcement for
certain resources, then they can uncomment these lines and then create the
.data file.

>Is the syntax of this file one URL per line?

Yes - see the Notes section of @pmFromFile -

>C2: Apart from ignoring certain URLs, could we specify some parameters
>to ignore (on some URLs) and on some HTTP methods?  i.e. ignore params
>if GET, but enforce if POST.

In ModSecurity v2.6 (available in trunk) we added the
SecRuleUpdateTargetById (and ctl:ruleUpdateTargetById) -

This capability will allow you to create chained rules with the
ctl:ruleUpdateTargetById to dynamically exclude ARGS from validation.

Of course in looking at these rules, we do need to assign rule IDs to them
for this to work.

>C3: Could the thresholds (50 and 100) be set near the top of the rule
>file, instead of within the SecAction of Step 1?

Well, it is the first configurable setting in the file.  What issue would
this solve?  We want to try and keep consistency with the CRS banner info
at the top and then I wanted to have a small intro section of text about
the detection.

>C4: Thresholds per host/site?
>C5: Some application owners may know all the allowable entry
>points.... it would be fantastic to have a way of defining that
>profile (in XML?) which the CRS profiler could
>   i) use as a base starting point, or
>   ii) use as a mandatory profile, or
>   iii) use to compare against when reporting discrepancies.
>  This would also help to group similar URLs (e.g. paths when URL
>rewriting is in use).

So if an organization actually had an XML map/profile of their entry
points right?  Hmm, you know what would be cool is to have something
similar to @validateSchema but called something like @validateProfile.
This would be able to parse an XML file that defines the allowable site
tree profile and then block/alert when request data that doesn't match.
That would be pretty cool.  :)  I guess the next step would be to come to
an agreed upon Site profile schema.  I think this is where your references
to Ivan's Recipes (http://www.modsecurity.org/projects/ppr/) and Chris
Bockermann's Web Profiles (http://jwall.org/web/policy/index.jsp) comes in
as we would need a standard method of defining an app profile.

>C6: Ability to modify model in C5 in real time (e.g. as URLs are added
>created when writing a new blog post, or pages are added in a CMS)

Agreed.  The way it is implemented currently with the CRS profiler rules,
however, is on a per-resource basis anyways.  So if you have a new
resource that is dynamically created by a CMS then it will begin to be

>C7: Ability for model in C5 to have the idea of "strict" and
>"allowable" parameter value definitions.  Example 1: we might want a
>phone number to be all numbers and spaces, but let's not get too upset
>with hyphens and parentheses.  Example 2: A trailing line break on a
>password could just be from copy & paste, not an attack.  So the
>profiler would then need to report whether a parameter value just met
>the allowable format, or failed that too.

This is referencing the AppSensor model of categorizing the event as
either Suspicious or Malicious.  As you can see in the rules, I have
mapped each rule to the corresponding AppSensor Detection point.  For
example -

My view is that these profiler rules only identify Suspicious events as
they don't match an expected learned profile.  In order to determine
Malicious events, the CRS should be run in Anomaly Scoring Mode
vs-anomaly-scoring-detection-modes.html) so that the transaction continues
into the other blacklist CRS rules to verify if there are XSS, SQLi attack
payloads, etc...  The profiler rules will increase an anomaly score and it
is set at a slightly lower severity level than attack rules (that have a
critical level) -


>M1: Accessing the same URL on multiple virtual hosts, using the same
>instance of ModSecurity (and with the same web browser) seems to
>create an aggregated profile rather than a profile per host (but I may
>be wrong):
>  http://www.example1.com/profiler.txt?site=2
>  http://www.example2com/profiler.txt?site=1&day=wednesday
>  Is there a way to profile each host separately?

Good point - I think that we need to expand the RESOURCE collection data
so that it can utilize the SecWebAppId directive data in the same way that
the setsid does so the collections are per-site.

>M2: Could (or Is?) the HTTP method be part of the profile so there
>could be different parameters for GET and POST for example.

I see your point as there may be different parameters that are valid based
on the Request Method.  We could do that with a new SecAction line at the
beginning like this -


The only issue I see is if a client sends a request to a resource with a
different REQUEST_METHOD, then it will create its own resource collection.
 So how would we validate which request methods are valid/allowed?  I
guess we could rely upon the app's response (HTTP Status Code of 405 -
Method Not Allowed) to let us know if the request was valid in this case
and then remove the collection.

>M3: Further data types (bit, SQL types, UUID, lists of integers, etc),
>and perhaps user-definable data types - both of these would be of more
>use if C5 is implemented.

I have a long list of other parameter types to add but I wanted to this
framework out for testing.

>M4: Ability to allow null values for parameters e.g. normally 0 or 1
>but could be NULL

Yeah, the RegEx length range checks are a but crude... Do you mean that a
param payload could be:

- foo.php?param=1
- foo.php?param=0
- foo.php?param

In these cases, the parameter named "param" might match both of these
length checks -

SecRule ARGS "^0$"
SecRule ARGS "^[1-5]$"

And then added to either the resource.args_length_check0_enforce or
resource.args_length_check1_enforce variable lists for enforcement.  It
would depend upon how often the param was empty/NULL.

>M5: Build result into anomaly scoring.

You mean something more than what is already there?  We already have two
levels of anomaly scoring built it -

- Contribution to the overall anomaly score

- Contribution to a "profiler" anomaly score which would allow a user to
add custom blocking level just for profiler check scores.

>Data Storage
>D1: Do you have any references for the format of this file, and of the
>profiler key records, so we can try to extract data and convert it
>into other formats?

You mean the data stored in the RESOURCE collections?  If so, I would
recommend using Christian Bockermann's jwall-tools java app so you can
view collection data outside of the Debug log -
https://www.jwall.org/tools/jwall-tools.jsp.  Perhaps Chris could update
the tools to extract RESOURCE collection data and translate it to an XML

Another cool idea would be for the jwall-tools to be able to actually
manipulate the RESOURCE collection data!  That would allow for some manual
tweaking of a profile rather than having to just delete and relearn it.

>Syndication of result
>S1: Could the "result" be added as another X-WAF-... Header

They should already be added to the default list of match events.  But if
you want a separate one, we could do that.

>S2: This one isn't Profiling-specific, but currently the X-WAF-
>headers are added to the request... could there be a method to pass
>data on by URL (e.g. like CSP reporting mechanism), so that say even
>if something is blocked it is possible to inform other devices of the
>event?  The data sent could be configurable, but might include what's
>currently in the X-WAF- headers, plus the original header content.

Just so I understand - we would block the original request but then send a
new HTTP request out from the WAF to other security devices with the X-WAF
data as URL parameters?  You should be able to do that by using the
"exec:" action and just write a script that will use something like
curl/wget to send the HTTP request and populate the data you need.  I will
test this out.

Thanks for the awesome feedback Colin!


>On 18 February 2011 15:11, Ryan Barnett <RBarnett at trustwave.com> wrote:
>> On 2/18/11 9:52 AM, "Lucas Ferreira" <listas at sapao.net> wrote:
>>>Hello Ryan,
>>>after reading your blog post, I still have some doubts:
>>>How does the profiling work with regard to restarting the web server?
>>>Are the collections saved to disk?
>> Yes - the data is saved to persistent storage files on disk.  This would
>> keep the data across Apache restarts.
>>>How can the administrator restart the profiling process? Is it
>>>possible to selectively restart profiling based on host, URL or other
>>>application identifier?
>> Great question.  We actually were talking a bit about some options for
>> profile management.  One option that we are thinking of would be to use
>> dynamic updates based on access requests from "trusted" clients.  Take a
>> look at the previous blog post on GeoIP data where we talk about dynamic
>> updates -
>> ation-data.html
>> The idea would be that if you wanted to remove/relearn a profile for a
>> URL, you could have an authorized client send a specific request to that
>> URL.  Based on the source IP, UA + Parameter (maybe
>> ?relearn_profile=true), then the saved resource collection data for that
>> URL would be removed and the relearning would commence.  There are some
>> other ideas that we we need to test which could possibly include
>> an increase in alerts across multiple clients as this might indicate an
>> application change and could initiate an auto-relearn.
>> This is something that we are planning to add in future versions.
>> Please keep the comments coming!  We want people to test so that we can
>> improve the capabilities.
>> Cheers,
>> Ryan
>>>On Thu, Feb 17, 2011 at 16:27, Ryan Barnett <RBarnett at trustwave.com>
>>>> Hello everyone,
>>>> This email will most likely come as very welcomed news to most of you.
>>>>We have just released a ruleset to the OWASP CRS that implements a
>>>>framework for real-time application profiling -
>>>> This initial version of the rules has the ability to profile and
>>>>enforce the following on a per-resource basis:
>>>>  *   Request Method(s)
>>>>  *   Number of Parameters
>>>>  *   Parameter Names
>>>>  *   Parameter Length Ranges
>>>>  *   Parameter Types - numeric or alpha
>>>> Please test out this ruleset and provide feedback on the OWASP CRS
>>>> Cheers,
>>>> Ryan
>>>> ________________________________
>>>> This transmission may contain information that is privileged,
>>>>confidential, and/or exempt from disclosure under applicable law. If
>>>>are not the intended recipient, you are hereby notified that any
>>>>disclosure, copying, distribution, or use of the information contained
>>>>herein (including any reliance thereon) is STRICTLY PROHIBITED. If you
>>>>received this transmission in error, please immediately contact the
>>>>sender and destroy the material in its entirety, whether in electronic
>>>>or hard copy format.
>>>> _______________________________________________
>>>> Owasp-modsecurity-core-rule-set mailing list
>>>> Owasp-modsecurity-core-rule-set at lists.owasp.org
>>>Homo sapiens non urinat in ventum.
>> _______________________________________________
>> Owasp-modsecurity-core-rule-set mailing list
>> Owasp-modsecurity-core-rule-set at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-modsecurity-core-rule-set
>Owasp-modsecurity-core-rule-set mailing list
>Owasp-modsecurity-core-rule-set at lists.owasp.org

This transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format.

More information about the Owasp-modsecurity-core-rule-set mailing list