• Forums
• Directory
• Blogs
• Glossary

Definitions of Robots.txt

• This is a text file in the root directory of a site that contains a set of instructions for the search engine spiders. It can list which search engine spiders are allowed to access and crawl the site, and which parts of the site the spiders should exclude. If there is no robots.txt file on a site the search engine spiders will crawl any of the pages on the site. This definition of Robots.txt contributed by jimolenbush.