See below Matt&#39;s script to use google to find robots.txt.<br><br>Next step on this script is to expand it to download a bund of those robots.txt and do an analysis on them (would be interresting to come up with some rules for expected and not expected robots.txt content)<br clear="all">
<br>Dinis Cruz<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Matt Parsons</b> <span dir="ltr">&lt;<a href="mailto:mparsons1980@gmail.com">mparsons1980@gmail.com</a>&gt;</span><br>
Date: 8 December 2010 23:09<br><br><br>Today I discuss a simple script that you can write in 02 to find robots.txt.   As we all know sometimes these scripts have sensitive information in them.   Feel free to comment or e-mail me.   02 is a free platform and an OWASP project.   <br>

<br><br><a href="http://parsonsisconsulting.wordpress.com/2010/12/08/how-to-find-robots-txt-with-02/" target="_blank">http://parsonsisconsulting.wordpress.com/2010/12/08/how-to-find-robots-txt-with-02/</a><div><div></div>
<div class="h5"><br><br><div class="gmail_quote"><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"><p>Matt Parsons, MSM, CISSP<br>315-559-3588 Blackberry<br>
817-294-3789 Home office <br>&quot;Do Good and Fear No Man&quot;  <br>Fort Worth, Texas<br>A.K.A The Keyboard Cowboy<br>

<a href="mailto:mparsons1980@gmail.com" target="_blank">mailto:mparsons1980@gmail.com</a><br><a href="http://www.parsonsisconsulting.com/" target="_blank">http://www.parsonsisconsulting.com</a><br><a href="http://www.o2-ounceopen.com/o2-power-users/" target="_blank">http://www.parsonsisconsultingblog.com   </a><br>


<a href="http://www.linkedin.com/in/parsonsconsulting" target="_blank">http://www.linkedin.com/in/parsonsconsulting</a><br><a href="http://www.vimeo.com/8939668" target="_blank">http://www.vimeo.com/8939668</a><br><a href="http://twitter.com/parsonsmatt" target="_blank">http://twitter.com/parsonsmatt</a><br>


 <br> </p>
<p> </p>
<p> </p><br>
</blockquote></div><br><br clear="all"><br>-- <br><p>Matt Parsons, MSM, CISSP<br>315-559-3588 Blackberry<br>817-294-3789 Home office <br>&quot;Do Good and Fear No Man&quot;  <br>Fort Worth, Texas<br>A.K.A The Keyboard Cowboy<br>

<a href="mailto:mparsons1980@gmail.com" target="_blank">mailto:mparsons1980@gmail.com</a><br><a href="http://www.parsonsisconsulting.com/" target="_blank">http://www.parsonsisconsulting.com</a><br><a href="http://www.o2-ounceopen.com/o2-power-users/" target="_blank">http://www.parsonsisconsultingblog.com   </a><br>

<a href="http://www.linkedin.com/in/parsonsconsulting" target="_blank">http://www.linkedin.com/in/parsonsconsulting</a><br><a href="http://www.vimeo.com/8939668" target="_blank">http://www.vimeo.com/8939668</a><br><a href="http://twitter.com/parsonsmatt" target="_blank">http://twitter.com/parsonsmatt</a><br>

 <br> </p>
<p> </p>
<p> </p><br>
</div></div><br>_______________________________________________<br>
Owasp-codereview mailing list<br>
<a href="mailto:Owasp-codereview@lists.owasp.org">Owasp-codereview@lists.owasp.org</a><br>
<a href="https://lists.owasp.org/mailman/listinfo/owasp-codereview" target="_blank">https://lists.owasp.org/mailman/listinfo/owasp-codereview</a><br>
<br></div><br>