Re: what is this robots.txt
I believe it's a supplemental file that you include in your root folder that includes the pages that you do not want the spiders to include in in their crawl. I don't want any of my pages excluded so I don't use this file and am not sure how to format it. Someone please correct me if I'm wrong on this one.
|