Robot follow command

[Nashvegas]

I have seen a couple of different ways to write robot follow and index commands. Here is what I am currently using: <meta name="robots" content="index, follow">

Do you think that putting this on every page is the best way to have G and Y index and rank your site? Have you found a better way? Thank you in advance!

[ResaleBroker]

IMO, a "robots.txt" file is the method of choice. There are plenty of tutorials on the subject.

Here are a few links:

• http://www.robotstxt.org/wc/robots.html
• http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

[sarahk]

robots.txt is used by a greater number of bots, the meta tags aren't well respected.

google does allow

<a href="whatever.php" rel="nofollow">whatever</a>

on the page but I wouldn't count on the others.

Sarah

[kensmith]

I have tested the on page no follow tag for some internal links that had duplicate content (canned stuff) and they got removed from G's index within a couple of weeks. No point in having duplicate content being indexed, the info is just there as filler for my customers.