[Owasp-appsensor-project] World Summit - AppSensor Results

John Melton jtmelton at gmail.com
Sat Mar 5 22:51:34 EST 2011


Colin,
Wow - a lot to digest, but fantastic ideas and detail :>.  I'm definitely
going to look into this more closely, but wanted to get a response out
quickly.  My short response would be yes and no (I love that response :>).
Basically, I would agree 98 % of apps probably write to a DB, and I also
agree that a mechanism like this would be the most logical in my opinion,
but not everyone does everything the same way.
I'm ruminating on the idea a bit more, but I'm thinking of refactoring the
"logging" to be a bit more simple, at least logically.  I'd like to get it
to the point where we have the intrusion object self-contained with all the
necessary data (several data points you have below are missing in the
current implementation) and that gets passed to some output/response
mechanism.  From there we should have a collection of handlers.  These
handlers logically look the same (follow something like the command pattern)
and get passed an intrusion.  We'll have a standard DB style handler, maybe
a set for different DB's, and a log handler, and a CEF handler, etc.  Since
we're building a "framework", we have to make it easy enough for folks to
plug in their system, or at least see how it should plug in.
I've not coded much on appsensor lately, and am looking back now to try to
analyze (with fresher eyes) some weak spots that can be worked a little
better. I'll try to keep everyone posted as I make any progress.
As always, if anyone on the list wants to jump into the java piece of the
project, ping me and I'll try to get you setup.
Thanks,
John

On Sat, Mar 5, 2011 at 5:40 AM, Colin Watson <colin.watson at owasp.org> wrote:

> John
>
> Regarding logging, I have found it better to set up generic
> application logging (into a DB), and then separately look at the
> exporting/syndicating a subset of that.
>
> In a logging implementation last year, I created a table for the
> security event log (not just attacks and other appsensor events)
> called `log_security` and a separate table for larger chunks of data
> (e.g. response bodies) if it was useful to record something more
> `log_security_details` :
>
> CREATE TABLE IF NOT EXISTS `log_security` (
>  `id_log` int(10) unsigned NOT NULL AUTO_INCREMENT,
>  `log_timestamp` datetime NOT NULL,
>  `severity` tinyint(3) unsigned NOT NULL DEFAULT '7',
>  `event_timestamp` datetime NOT NULL,
>  `event_type` varchar(25) CHARACTER SET latin1 DEFAULT NULL,
>  `action` varchar(25) CHARACTER SET latin1 NOT NULL,
>  `object` varchar(25) CHARACTER SET latin1 NOT NULL,
>  `result_status` varchar(7) CHARACTER SET latin1 DEFAULT NULL,
>  `reason` varchar(255) CHARACTER SET latin1 DEFAULT NULL,
>  `http_status_code` smallint(5) unsigned DEFAULT NULL,
>  `event_confidence` tinyint(3) unsigned NOT NULL DEFAULT '100',
>  `location` varchar(100) DEFAULT NULL,
>  `host` varchar(45) CHARACTER SET latin1 NOT NULL,
>  `service_name` varchar(25) CHARACTER SET latin1 NOT NULL,
>  `service_port` smallint(5) unsigned NOT NULL DEFAULT '80',
>  `service_protocol` varchar(10) CHARACTER SET latin1 NOT NULL DEFAULT
> 'http',
>  `http_method` varchar(10) CHARACTER SET latin1 DEFAULT NULL,
>  `entry_point` varchar(100) CHARACTER SET latin1 DEFAULT NULL,
>  `request_number` int(10) unsigned DEFAULT '0',
>  `source` varchar(45) CHARACTER SET latin1 NOT NULL COMMENT 'IP address',
>  `user_identity` int(10) unsigned NOT NULL DEFAULT '0' COMMENT
> 'id_person or 0 for unauthenticated',
>  `http_user_agent` varchar(120) CHARACTER SET latin1 DEFAULT NULL,
>  `user_fingerprint` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
>  `hash` varchar(40) CHARACTER SET latin1 DEFAULT NULL,
>  PRIMARY KEY (`id_log`),
>  KEY `Index_2` (`severity`),
>  KEY `Index_3` (`request_number`),
>  KEY `Index_4` (`user_identity`)
> ) ENGINE=InnoDB  DEFAULT CHARSET=utf8;
>
> CREATE TABLE IF NOT EXISTS `log_security_details` (
>  `id_log` int(10) unsigned NOT NULL,
>  `details` text COLLATE latin1_general_ci NOT NULL,
>  PRIMARY KEY (`id_log`)
> ) ENGINE=MyISAM DEFAULT CHARSET=latin1 COLLATE=latin1_general_ci;
>
> I also needed:
>
> CREATE TABLE IF NOT EXISTS `log_request_counter` (
>  `counter` int(10) unsigned NOT NULL DEFAULT '1',
>  `modified` datetime DEFAULT NULL,
>  PRIMARY KEY (`counter`)
> ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
>
> INSERT INTO `log_request_counter` (`counter`, `modified`) VALUES
> (47, '2010-10-27 16:31:44');
>
> I also had another table `log_data`  used to track changes/deletions
> of individual database fields, but that isn't relevant here.
>
> In the main log,
>
> - `id_log` is an auto-incrementing ID
>
> - `log_timestamp` is determined when the log entry is written - not
> when the event was detected
>
> - `severity` is
>      0 (Emergency/Application unavailable for all users)
>      1 (Alert/Function unavailable for all users),
>      2 (Critical/Function or application unavailable to a single user),
>      3 (Error/Other security events not included in codes 0, 1, 2 or 4),
>      4 (Warning/A security event but user allowed to continue),
>      5 (Notice: normal but significant condition)
>      6 (Information/Normal user behaviour)
>
> - `event_timestamp` is the event date/time
>
> -  `event_type` is a broad description e.g. "Input Validation",
> "Authentication", "Session Management", "Output Validation"
>
> -  `action` was the original intended purpose of the request e.g. Log
> In, Refresh Session ID, Log Out, Request
>
> -  `object` is the affected component or other object (user account,
> data resource, file) e.g. URL, Session ID, User Account
>
> - `result_status` is whether the ACTION aimed at the OBJECT was
> successful e.g. Success, Fail, Defer
>
> - `reason` is why the status above occurred e.g. User not
> authenticated in database check ..., Incorrect credentials
>
> - `http_status_code` is the status code returned to the user (often
> 200 or 301, need to imagine many log entries here for a single
> request)
>
> - `event_confidence` is a percentage 0-100...confidence the event
> identified occurred as stated above - often 100% for whitelist input
> validation, but maybe less than that for data from a WAF about
> possible XSS attack
>
> - `location` is where event was detected - script or component (not the
> URL)
>
> - `host` e.g. www.example.com
>
> - `service_name` e.g. "Colin's App"
>
> - `service_port` e.g. 443
>
> - `service_protocol` e.g. http
>
> - `http_method` e.g. GET, POST
>
> - `entry_point` a URL e.g. /cw/Bench.php
>
> - `request_number` is a unique integer for every single request
> received by the application, and will be shared by a number of ecent
> log entries, so they can be correclated with each other i.e. one
> rqeuest may have multiple input validation errors
>
> - `source` varchar(45) CHARACTER SET latin1 NOT NULL COMMENT 'IP address',
>
> - `user_identity` might be a user ID, username or perhaps 0 for
> unauthenticated',
>
> - `http_user_agent` is HTTP_USER_AGENT
>
> -  `user_fingerprint` is a SHA!1 hash of user agent/request properties
>
> sha1(substr(trim($_SERVER['REMOTE_ADDR']),0,7).trim($_SERVER['HTTP_USER_AGENT']).trim($_SERVER['HTTP_ACCEPT_ENCODING']).trim($_SERVER['HTTP_ACCEPT_LANGUAGE']));
>
> - `hash` a SHA1 hash of a salt concatenated with the hash of the
> immediately previous log entry and the log record above (including
> id_log) for integrity checking
>
> the only other thing I had wondered about adding was a field for which
> (e.g. type-identifier like RE3-2001) AppSensor detection point fired.
> This may be NULL for some entries.  Then perhaps also what responses
> ocuured (e.g. a list A,F,G or NULL).  That way it's easier to
> monitor/tune AppSensor.  Or possibly those two should be logged in
> another DB table..
>
>  id_log
>  appsensor_detectionpoint
>  appsensor_response (allow NULL)
>
> allowing for more than one record per id_log
>
> Some example log entries are:
>
> INSERT INTO `log_security` (`id_log`, `log_timestamp`, `severity`,
> `event_timestamp`, `event_type`, `action`, `object`, `result_status`,
> `reason`, `http_status_code`, `event_confidence`, `location`, `host`,
> `service_name`, `service_port`, `service_protocol`, `http_method`,
> `entry_point`, `request_number`, `source`, `user_identity`,
> `http_user_agent`, `user_fingerprint`, `hash`) VALUES ...
>
> (1205, '2010-10-08 08:39:29', 3, '2010-10-08 08:39:29', 'Input
> Validation', 'Request', 'URL', 'Fail', 'GET parameter ''dialogID'' is
> not permitted', 400, 100, 'includes:parameters-check-allowed.php',
> 'example.com', 'CW Bench', 80, 'http', 'GET', '/cw/bench.php', 832,
> 'xxx.xxx.xxx.xxx', 49, 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT
> 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; InfoPath.2; .NET CLR
> 3.0.3072', 'ed69a08f107c700f59c265198228f7968ac57e70',
> '26e9a8d4134180283cf04b533cd7de28c1c4545c'),
>
> (1424, '2010-10-08 16:50:30', 3, '2010-10-08 16:50:30', 'Input
> Validation', 'Request', 'URL', 'Fail', 'GET parameter
> ''script>ALERT_1__/script>'' is not permitted', 400, 100,
> 'includes:parameters-check-allowed.php', 'example.com', 'CW Bench',
> 80, 'http', 'GET', '/cw/bench.php', 2408, 'xxx.xxx.xxx.xxx', 0,
> 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.3
> (KHTML, like Gecko) Chrome/6.0.472.63 Safari/534.3',
> '98309085b4e63d8a1f3ef898a6f05e1f83affc97',
> 'ec3dfae274a7330289c7b618a292612a891f7a3e'),
>
> (2014, '2010-10-18 10:12:11', 3, '2010-10-18 10:12:11', 'Session
> Management', 'Check Authorised', 'Session ID', 'Fail',
> '$_SESSION[''AUTHORIZED''] does not exist', 301, 100,
> 'includes:check-authorised.php', 'example.com', 'CW Bench', 80,
> 'http', 'GET', '/cw/Bench.php', 103, 'xxx.xxx.xxx.xxx', 0,
> 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.10)
> Gecko/20100914 Firefox/3.6.10',
> '1e8d88986cc07e586acdd0c39f92728db6df8c06',
> '18d2114afb34211390851e6435149f220baedc80'),
>
> (3609, '2010-10-26 18:11:38', 5, '2010-10-26 18:11:38', 'Session
> Management', 'Log out', 'User Account', 'Success', '', 301, 100,
> 'terminate.php', 'example.com', 'CW Bench', 80, 'http', 'POST',
> '/cw/terminate.php', 82, 'xxx.xxx.xxx.xxx', 0, 'Mozilla/5.0 (Windows;
> U; Windows NT 5.1; en-GB; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11',
> '8c01499deb24a67ac60ae49aef867bcd7c434b28',
> 'd89617515fadca0366cc74ede58855a9e1733066'),
>
>
> (3663, '2010-10-27 16:30:17', 5, '2010-10-27 16:30:17',
> 'Authentication', 'Log In', 'User Account', 'Success', 'Successful log
> in', 301, 100, 'gegin.php', 'example.com', 'CW Bench', 80, 'http',
> 'POST', '/cw/first.php', 34, 'xxx.xxx.xxx.xxx', 46, 'Mozilla/5.0
> (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.11) Gecko/20101012
> Firefox/3.6.11 ( .NET CLR 3.5.30729)',
> '511ceb082b2e8634cec7e160802eab9d15c185b4',
> 'ce45be9a7654b9037e1922f9a75318c4182a89cb'),
>
>
> (3670, '2010-10-27 16:30:34', 3, '2010-10-27 16:30:34', 'Input
> Validation', 'Request', 'URL', 'Success', 'Attack - total impact 22
> (xss, csrf, sqli, id, lfi, rfe)', 200, 50, 'includes:phpids.php',
> 'example.com', 'CW Bench', 80, 'http', 'POST',
> '/cw/jsonTemplate1.php', 41, '86.162.235.215', 46, 'Mozilla/5.0
> (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.11) Gecko/20101012
> Firefox/3.6.11 ( .NET CLR 3.5.30729)',
> '511ceb082b2e8634cec7e160802eab9d15c185b4',
> 'f4052242441f4896e02cf6c1678240c4419eb502'),
>
>
>
> An example of a related additional details entry is data from PHPIDS:
>
> INSERT INTO `log_security_details` (`id_log`, `details`) VALUES
> (1219, 'Total impact: 56<br/>\nAffected tags: xss, csrf, id, rfe,
> lfi<br/>\n<br/>\nVariable: REQUEST.formtitle | Value: XXXXX
> <br/>\nImpact: 7 | Tags: xss, csrf, id, rfe, lfi<br/>\nDescription:
> Detects unknown attack vectors based on PHPIDS Centrifuge detection |
> Tags: xss, csrf, id, rfe, lfi | ID: 67<br/>\n<br/>\nVariable:
> REQUEST.formdescription | Value: XXXXX<br/>\nImpact: 7 | Tags: xss,
> csrf, id, rfe, lfi<br/>\nDescription: Detects unknown attack vectors
> based on PHPIDS Centrifuge detection | Tags: xss, csrf, id, rfe, lfi |
> ID: 67<br/>\n<br/>\nVariable: REQUEST.formlevel5 | Value:
> XXXXX.<br/>Impact: 7 | Tags: xss, csrf, id, rfe, lfi | ID:
> 67<br/>\n<br/>Centrifuge detection data<br/>  Threshold: 3.49<br/>
> Ratio: 2.2857142857143<br/><br/>\n'),
>
>
>
> On 3 March 2011 04:51, John Melton <jtmelton at gmail.com> wrote:
> > These ideas sound great. Responses inline below.
> >
> > Also, I asked on this list a while back, and I'll try again as there may
> be
> > new folks on, does anyone have a doc that describes the common event
> format
> > (CEF)? I would like to write something that will log to this format, but
> I
> > don't have any tools that read it, or any documentation on how to write
> it.
> > This could be a useful log format, as I understand it can be fed into
> SEIM
> > tools. Any pointers here would be helpful.  Some of the questions and
> > comments I've heard seem to point to the fact that supporting various
> output
> > formats is a definite positive for this project going forward.
> >
> > Thanks,
> > John
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.owasp.org/pipermail/owasp-appsensor-project/attachments/20110305/5bf5fe13/attachment-0001.html 


More information about the Owasp-appsensor-project mailing list