Re: what is this robots.txt
Google wants to know that it's actually allowed to crawl your web site before doing so. It doesn't trust you on your word... it wants to see that it is allowed to spider your site. Google may also ask you to post a "key" to your web site for enhanced functionality.
Robots.txt is well documented throughout the web.
|