[Owasp-modsecurity-core-rule-set] Call for Assistance: ModSecurity/CRS Event Data Statistics

Ryan Barnett RBarnett at trustwave.com
Thu Oct 7 10:25:27 EDT 2010

I wanted to say thanks to those of you who have sent in some stats!  I see that some users are already using Christian Bockermann’s updated jwall-tools.jar file to gather stats which is cool.

We still needs more stats sent in though, so if you have a minute, please send them in.

Thanks again for your help.


On 10/4/10 4:49 PM, "Ryan Barnett" <RBarnett at trustwave.com> wrote:

Greetings everyone,
I am asking for some assistance from the ModSecurity user-base.  I am working on a project and need to gather some high level statistics of ModSecurity event data.  Our long-term goal is to eventually have a statistical reporting utility that ModSec users can run through cron or something to report data in semi-realtime batches.  We can then post this data onto the ModSecurity.org website.  This data would only be an anonymous count of the ModSecurity/CRS Event message data (SQL Injection Attack, Invalid character in request, etc....).  It will not include any sensitive data identifying web sites, users or their data.

In the short-term, what I am asking for you all to do to help with this initial effort, is to simply run the following command pipeline against their saved ModSecurity audit log data.  The audit log can be either serial or concurrent.  All you need to do is to traverse into your logs directory (example - /usr/local/apache/logs).  This directory needs to be the directory that holds either the serial modsec_audit.log file or where the concurrent SecDataDir directory is located.  Next, execute the following command (you may need to use sudo or something if you do not have the correct permissions to view the file) -

$egrep -R '^Message\:' . | awk -F' \\[msg ' '{ print $2 }' | awk -F'"' '{print $2}' | egrep -v '^(Inbound Anomaly Score|$)' | sort | uniq -c | sort -nr

This command will extract out the ModSecurity Message lines from the audit_log data and then sort and unique the ModSecurity event message “msg” data from all of the events.  The output format should be similar to this -

3809 Rogue web site crawler
  3809 Request Indicates a Security Scanner Scanned the Site
  3808 Request Missing an Accept Header
   795 Cross-site Scripting (XSS) Attack
   538 Detects very basic XSS probings
   478 Detects possibly malicious html elements including some attributes
   474 Detects obfuscated script tags and XML wrapped HTML
   434 Detects specific directory and path traversal
   421 XSS Attack Detected
   396 IE XSS Filters - Attack Detected
   359 Remote File Access Attempt
   324 Possible XSS Attack Detected - HTML Tag Handler
   318 Detects basic directory traversal
   274 Detects etc/passwd inclusion attempts
   155 Host header is a numeric IP address
   155 finds html breaking injections including whitespace attacks
   145 Detects JavaScript object properties and methods
   116 Detects basic SQL authentication bypass attempts 2/3
   112 Detects common XSS concatenation patterns 1/2
    96 Detects classic SQL injection probings 2/2
    94 Detects self-executing JavaScript functions
    84 Detects possible includes and typical script methods
    83 Invalid character in request
    69 Restricted Character Anomaly Detection Alert - Repetative Non-Word Characters
    54 System Command Injection
    32 Detects common comment types
    31 Detects possible includes, VBSCript/JScript encodeed and packed functions
    30 Comment Evasion Attempt
    27 System Command Access
    25 More than 3 times special encode  Error
    25 Detects JavaScript location/document property access and window access obfuscation
    18 GET or HEAD requests with bodies
    18 Detects basic SQL authentication bypass attempts 3/3
    14 Restricted Character Anomaly Detection Alert - Total # of special characters exceeded
    13 Detects JavaScript language constructs
    12 HTTP protocol version is not allowed by policy
    10 Detects data: URL injections, VBS injections and common URI schemes
     9 Detects classic SQL injection probings 1/2
     8 Input Validation  Error
     8 Detects code injection attempts 3/3
     7 Detects chained SQL injection attempts 1/2
     6 Invalid HTTP Request Line
     6 finds attribute breaking injections including whitespace attacks
     5 Method is not allowed by policy
     4 SQL Injection Attack
     4 Detects obfuscated JavaScript script injections
     4 Detects halfwidth/fullwidth encoded unicode HTML breaking attempts
     4 Detects common function declarations and special JS operators
     4 Detects code injection attempts 2/3
     3 Request Missing a Host Header
     3 Detects MySQL comment-/space-obfuscated injections
     3 Detects basic obfuscated JavaScript script injections
     2 Unicode Full/Half Width Abuse Attack Attempt
     2 Remote File Inclusion Attack
     2 Hexadecimal Charcode Pattern Found
     2 finds unquoted attribute breaking injections
     2 Detects the IE octal, hex and unicode entities
     2 Detects nullbytes and other dangerous characters
     2 Detects JavaScript string properties and methods
     2 Detects basic SQL authentication bypass attempts 1/3
     1 URL Encoding Abuse Attack Attempt
     1 Request Missing a User Agent Header
     1 Possible RegEx DoS Payload
     1 Detects basic XSS DoS attempts
     1 Blind SQL Injection Attack
     1 Backdoor access

Once you have run this against your saved logs, please send them to me in email to the following email address – security at modsecurity.org.  If you could, also please specify the following in the email body -

 1.  How many websites ModSecurity is monitoring, and
 2.  The date range of the data included in the stats report (we are looking for all data from 2010).

Thanks for your help in this research effort.

Ryan Barnett
ModSecurity Community Manager
OWASP ModSecurity Core Rule Set Project Lead

More information about the Owasp-modsecurity-core-rule-set mailing list