[Owasp-data-exchange-format] (lets do it!) Re: DEF Strawman characteristics

Tue Jul 26 08:33:58 EDT 2011

Hi Ferruh,

Be great to have your input on the draft XSD :)

I've uploaded examples from Burp and ZAP to
http://code.google.com/p/owasp-def/source/browse/#svn%2Ftrunk%2Fexamples
Anyone else able to upload other examples?
Email your Google acc id to me or Dinis to get commit access to this
project.
Or email then directly to me and I'll upload them.

I'm also getting the feeling that people would really be much happier with
XML/XSD rather than JSON.
I really dont have a problem with that, and its what a strawmans for.

Lets see if we can get a few more examples uploaded and if they are all XML
then that will settle it.
We can try to merge them to something sensible and go from there.

Re who decides what (like when 1.0 is declared to be final), I'm hoping we
can get a broad consensus on this.
At the very least I'd like the OK from all of the leaders of the main OWASP
projects involved.

Cheers,

Simon

On Mon, Jul 25, 2011 at 5:19 PM, Ferruh Mavituna <ferruh at mavituna.com>wrote:

> Dinis, as you said from
> https://www.owasp.org/index.php/Summit_2011/Open_letter_to_WebAppSec_Tool_and_Services_vendors:_Release_your_schemas_and_allow_automation you
> have plenty of data, first you upload them to code repository of DEF (*if
> it doesn't exist I'm project leader will create it and add you as
> contributor*).
>
> Then based on that we just need to come up with one example document. I
> suggested XSD because it's better than a lousy word/wiki document it can
> validate other documents, it's documentation and specification.
>
> Let's get the first XSD and sample XML up. Call it version 1.0 and that's
> it.
>
> I have been wanting to add support to the tools that are already
>> represented in this thread (ZAP, Netsparker), but when* I actually sit
>> down and tried to do it, where do it start? *
>
>
> From first hand experience when our customers or other products (such as
> Metasploit or Vulnerability Manager<http://vulnerabilitymanager.denimgroup.com/>)
> we just point them to our XSD and sample XML files and that's it.
>
> JSON sounds cool but at this point unnecessary, we should stick with the
> simplest and one solution. What benefits JSON provide over XML? Not much or
> maybe none in this scenario. Data size is simply doesn't matter, also I
> don't agree with any format such as OpenXML zipped stuff, it simply make
> things more complicated. If someone thinks XML is big let them 7zip it in
> their desktop, it's not producers or consumers problem. Let's have one thing
> and let's sure it's working, easy and can be consumed & produced by most
> different products.
>
> I'm also happy to work on the first XSD draft.
> *
> *
>
> On 25 July 2011 09:48, dinis cruz <dinis.cruz at owasp.org> wrote:
>
>> I really like the idea to use JSON as a way to store data,  which together
>> with a strong XSD/schemas should provide enough flexibility (REST interfaces
>> are great, but realistically they will be implemented by 'app sec
>> integration' tools/portals).
>>
>> *In terms of trying to get more 'feedback' and create a first pass at a
>> 'schema' I'm would advise against it.* We are now at exactly the moment
>> where (in my view) all past efforts to do this failed.
>>
>> We don't need more ideas, planning, XSD, etc...
>>
>> What we need is to start trying to make it work.
>>
>> Let's take baby steps. Lets:
>>  - collect the data/schemas that are available TODAY
>>  - come up with a couple use-cases to connect them
>>  - implement those use cases
>>
>> Just doing this ONCE would be enormously valuable.
>>
>> I would say that our first step should be to collect all schemas and
>> sample assessment files (or whatever the tools can produce) and put them all
>> in one place.
>>
>> This is what I tried to do with
>> https://www.owasp.org/index.php/Summit_2011/Open_letter_to_WebAppSec_Tool_and_Services_vendors:_Release_your_schemas_and_allow_automationand one of its key objectives was to kick-start the conversation and point
>> the direction (and see any tool vendor would complain spectacularly  (which
>> they didn't)) Now is the perfect time to implemented it.
>>
>> Take a look at what I did in O2 to add support to Fortify: Fortify «
>> OWASP O2 Platform Blog<http://o2platform.wordpress.com/category/tools/fortify/> which
>> is what needs to happen on the SourcCode Analysis land, and if you want a
>> Black-box use case, what about this one: *"Here are the links to a Spring
>> MVC app **
>> http://o2platform.wordpress.com/2011/07/15/visualizing-the-links-in-jpetstore-spring-mvc/
>> *<http://o2platform.wordpress.com/2011/07/15/visualizing-the-links-in-jpetstore-spring-mvc/>
>> * how can tool xyz consume it?"*
>>
>>
>> I have been wanting to add support to the tools that are already
>> represented in this thread (ZAP, Netsparker), but when* I actually sit
>> down and tried to do it, where do it start? *
>> *
>> *
>> I have a bunch of emails somewhere in my inbox that have the info that I
>> need, but what I (and every other tool developer that wants to implement
>> this)  need, is a central location to get the required files.
>>
>> In fact, I already have a bunch of sample files form a bunch of different
>> tools, so where do I upload them? (do we have a code repository for DEF?)
>>
>> Dinis Cruz
>>
>> On 25 July 2011 11:23, Daniel Brzozowski <daniel at brzozowski.biz> wrote:
>>
>>> Just throwing some ideas in the air :)
>>> anyway, it wouldn't be much effort to do both - XML and JSON, the most
>>> important thing is content but we need to decide which way we go... So maybe
>>> we should create a poll to get some feedback from people?
>>>
>>> --
>>> Regards,
>>> db
>>>
>>>
>>> On 25 July 2011 09:51, psiinon <psiinon at gmail.com> wrote:
>>>
>>>> I specified JSON in an attempt to keep things as simple as possible, but
>>>> I'll admit XML came a close second.
>>>> And Dafydd pointed out that XML cant handle null bytes unless you do
>>>> something like base 64 encoding.
>>>> But if the majority of people think XML is more suitable then I'll be
>>>> happy to go with that :)
>>>>
>>>> I know quite a few products already export data in various formats.
>>>> It would be _really_ useful if people could email this list with
>>>> examples.
>>>>
>>>> Many thanks,
>>>>
>>>> Simon
>>>>
>>>> On Sun, Jul 24, 2011 at 5:49 PM, Ferruh Mavituna <ferruh at mavituna.com>wrote:
>>>>
>>>>> I might not have the background in here, but I think REST should be
>>>>> completely considered as a completely separate project / extension.
>>>>> Secondly why not simply come up with an XSD (*which is completely
>>>>> designed for this purpose*) and leave it there, wouldn't that be
>>>>> enough?
>>>>>
>>>>> On 24 July 2011 12:05, psiinon <psiinon at gmail.com> wrote:
>>>>>
>>>>>>  Ah, yes, I should really explain the 'REST' interface a bit more:)
>>>>>> And, thinking about it, a few use cases wouldnt go amiss!
>>>>>>
>>>>>> My plan for the strawman was to define a hierarchical structure for
>>>>>> representing pentest data, covering all of the types of objects in the data
>>>>>> model (hosts, ports, sites (host:port), urls, issues, requests/responses).
>>>>>> However all of the objects will be optional, as it really depends on
>>>>>> whats generating it and how the data is going to be used.
>>>>>> So, you might want to export everything regarding a pentest session
>>>>>> from burp, in which case a massive file might be generated.
>>>>>> Or you might want to just export a single request to feed into
>>>>>> JBroFuzz.
>>>>>> The file format should allow these 2 extremes and pretty much anything
>>>>>> inbetween.
>>>>>> Products could either prompt the user as to what they want to export,
>>>>>> or in some cases just ignore the parts that dont apply.
>>>>>>
>>>>>> However I also really like the idea of defining an API, which I know
>>>>>> is not really in the spirit of a data format.
>>>>>> Maybe this could be a separate but related doc?
>>>>>>
>>>>>> To support the REST API products would expose a base URL, eg
>>>>>> http://localhost:1234/myproduct/owasp-def1.0/
>>>>>> There would be a standard set of REST endpoints under that which would
>>>>>> allow other products to retrieve selected objects.
>>>>>> So if your do a GET on (for example)
>>>>>> http://localhost:1234/myproduct/owasp-def1.0/sitemap
>>>>>> you would get back a summary of all of the URLs the product has
>>>>>> discovered.
>>>>>> You could then do a GET on
>>>>>> http://localhost:1234/myproduct/owasp-def1.0/reqresp?id=1234
>>>>>> to get the full request/response pair for that request using the app
>>>>>> specific id, which would have been returned with the URL details.
>>>>>> All of the data returned will (or course) be in DEF format.
>>>>>>
>>>>>> Not all products would support all objects, so the top level URL would
>>>>>> essentially list which objects (or endpoints) are supported.
>>>>>>
>>>>>> That a bit clearer?
>>>>>>
>>>>>> And, more importantly, should it be part of the DEF, related but
>>>>>> separate or am I being too ambitious and therefore should it be put on hold
>>>>>> or scrapped?
>>>>>>
>>>>>> Re the parameters and cookies - I was thinking that they would just be
>>>>>> part of the request/response data.
>>>>>> But I can see that sometimes it might be really useful to be able to
>>>>>> refer to them directly.
>>>>>> Definitely one to think about!
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> On Sat, Jul 23, 2011 at 5:01 PM, Martin Holst Swende <
>>>>>> martin.holst_swende at owasp.org> wrote:
>>>>>>
>>>>>>>  ----- Ursprungsmeddelande -----
>>>>>>> > I've updated the Strawman tab on the project page (
>>>>>>> >
>>>>>>> https://www.owasp.org/index.php/OWASP_Data_Exchange_Format_Project#tab=Strawman)
>>>>>>>
>>>>>>> > with some proposed characteristics:
>>>>>>> >
>>>>>>> >      - The format will be JSON (to make it as simple as possible)
>>>>>>>
>>>>>>> great!
>>>>>>>
>>>>>>> >      - Products can generate and/or consume DEF
>>>>>>> >      - Products will be able to generate DEF via a defined REST
>>>>>>> interface
>>>>>>>
>>>>>>> could you please explain this a bit? Not sure I understand...
>>>>>>>
>>>>>>> >      and/or simple files - products can choose
>>>>>>> >      - Products which consume DEF must support both REST and file
>>>>>>> options
>>>>>>> >      - There will be minimal security (but REST based services can
>>>>>>> limit
>>>>>>> > by IP    addr)
>>>>>>> >      - The data model will cover: hosts, ports, sites (host:port),
>>>>>>> urls,
>>>>>>> >      issues, requests/responses
>>>>>>>
>>>>>>> How about parameters? E.g if a certain param or field is vulnerable
>>>>>>> or should be exported as fuzzer-target? Cookies? Perhaps also http header
>>>>>>> fields.
>>>>>>>
>>>>>>> >      - Products can generate a subset of DEF, the level support
>>>>>>> will be
>>>>>>> >      described in the DEF
>>>>>>> >
>>>>>>> > What do you think??
>>>>>>> >
>>>>>>> > Let me know if I've been too terse!
>>>>>>> >
>>>>>>> > Simon
>>>>>>>
>>>>>>> sounds good so far! Keep it up!
>>>>>>> Regards,
>>>>>>> Martin from n900
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Owasp-data-exchange-format mailing list
>>>>>> Owasp-data-exchange-format at lists.owasp.org
>>>>>> https://lists.owasp.org/mailman/listinfo/owasp-data-exchange-format
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> .fm
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Owasp-data-exchange-format mailing list
>>>> Owasp-data-exchange-format at lists.owasp.org
>>>> https://lists.owasp.org/mailman/listinfo/owasp-data-exchange-format
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Owasp-data-exchange-format mailing list
>>> Owasp-data-exchange-format at lists.owasp.org
>>> https://lists.owasp.org/mailman/listinfo/owasp-data-exchange-format
>>>
>>>
>>
>> _______________________________________________
>> Owasp-data-exchange-format mailing list
>> Owasp-data-exchange-format at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-data-exchange-format
>>
>>
>
>
> --
> .fm
>
> _______________________________________________
> Owasp-data-exchange-format mailing list
> Owasp-data-exchange-format at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-data-exchange-format
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.owasp.org/pipermail/owasp-data-exchange-format/attachments/20110726/4b023d28/attachment-0001.html 