[Owasp-appsensor-project] Detection or protection from web scraping

Colin Watson colin.watson at owasp.org
Fri Aug 16 14:34:01 UTC 2013


Junior

Thank you for the suggestions. I have updated UT2 on the wiki page
(and the equivalent text in the draft v2 Guide book) to mention
content scraping explicitly.

Regards

Colin


On 12 July 2013 05:16, Junior Lazuardi <junior.lazuardi at owasp.org> wrote:
> Hi Dennis and Ryan,
>
> thanks for the info, i see some of them are covered already, also on
> AppSensor cheat sheet
> Just curious that it's not "easy" to search on scraping explicitly, compared
> to those commercial ones.
> People might not be informed of it.
>
> Anyway, is scraping classified as an attack?
> and websites allowing scraping to be having vulnerabilities?
>
> a couple of search results at www.owasp.org
>
> https://www.owasp.org/index.php/AppSensor_Cheat_Sheet
> "Unacceptable behavior could include unauthorized scraping of content,
> searching for vulnerabilities, and attempts undertake fraud."
>
> https://www.owasp.org/index.php/Los_Angeles/2010_Meetings/November_24
> "But signatures have become less effective at detecting threats over time,
> and aren’t sufficient to address the sophisticated abusive behavior that
> large, publicly exposed Web applications are subject to, including page
> scraping, logic abuse, malicious automation, phishing, and malware
> distribution."
>
>
> maybe add "content scraping / abuse" in UT2? or unnecessary (inclusive)?
>
> UT2: Speed of Application Use
> The speed of requests from a user indicates that an automated tool is being
> used to access the site. The use of a tool undertaking a high number of
> requests quickly may indicate reconnaissance for an attack or attempts to
> identify vulnerabilities in the site.
>
> Thanks,
>
> Junior
>
>
> On Thu, Jul 11, 2013 at 7:26 PM, Ryan Barnett <ryan.barnett at owasp.org>
> wrote:
>>
>> Web Scraping would be covered by this AppSensor Detection Point -
>>
>> https://www.owasp.org/index.php/AppSensor_DetectionPoints#UT2:_Speed_of_Application_Use
>>
>> -Ryan
>>
>> From: Junior Lazuardi <junior.lazuardi at owasp.org>
>> Date: Thursday, July 11, 2013 4:26 AM
>> To: <owasp-appsensor-project at lists.owasp.org>
>> Subject: [Owasp-appsensor-project] Detection or protection from web
>> scraping
>>
>> Hello project members,
>>
>> I'm curious about web scraping, and can't find many on OWASP website, or
>> on available projects.
>>
>> Some characteristics of web scraping might have been covered already in
>> AppSensor Detection Points, but the word/phrase itself is so rarely
>> mentioned.
>> I found that some commercial products (WAF) sell web scraping protection
>> as a feature, and they even have specific whitepaper on it.
>>
>> Please let me know what you think of it, and whether it fits on appsensor
>> document.
>>
>> best regards,
>>
>> Junior
>>
>>
>>
>> _______________________________________________ Owasp-appsensor-project
>> mailing list Owasp-appsensor-project at lists.owasp.org
>> https://lists.owasp.org/mailman/listinfo/owasp-appsensor-project
>
>
>
> _______________________________________________
> Owasp-appsensor-project mailing list
> Owasp-appsensor-project at lists.owasp.org
> https://lists.owasp.org/mailman/listinfo/owasp-appsensor-project
>


More information about the Owasp-appsensor-project mailing list