Yahoo crawler/bot

[SEO Jason]

thanks for the great robots.txt file Kyle.

[Memphis]

These are bots that you would want to stop, why? Becasuse they are not considered useful?

Does REW do this for their sites?

Also, I noticed you added google picture bot and some Microsaft bots? You wouldn't want google indexing the photos of your lovely town? or of your beautiful face?

Just trying to figure things out..

[kyle422]

Also, I noticed you added google picture bot and some Microsaft bots? You wouldn't want google indexing the photos of your lovely town? or of your beautiful face?

What happens is people use your pictures and hotlink them off your site. Let's say someone is looking for a beach picture on Google images to use on their Myspace site. Now that picture is on their Myspace page. Let's say 20 other people like that picture and use it on their site. That number can grow drastically in just a matter of days. Myspace has more unique visits and pageloads than Google or any other website. So if you have any image that is being used on potentially thousands of pages on Myspace you could lose some serious bandwidth to hotlinking.
If you add this bit of code to your .htaccess you can prevent hotlinking (or I assume most hotlinking).

RewriteRule \.(jpe?g|gif|png)$ folder/no_hotlink.jpg [R,L]

Change the folder name to wherever you store your images on your server.

disclaimer: I'm not an expert at coding or coding theory

[Memphis]

Thanks for the info.

Does REW do this for the sites that they host?

[webmaster]

Memphis, we will upload any robots.txt file you like or put in any code in the .htaccess file. By default we dont upload robots.txt as that file Kyle posted is overly cautious, we dont have a major problem on our sites with those bots (Many of which I think dont even go out much any more) - however if you want that file on your site, just make a request in the template support forum. I dont even have it at REW as I dont see those bots (Or most of them) in my log files for the past year. I do however use it to block our beta folders. www.realestatewebmasters.com/robots.txt

[Memphis]

Thanks again. If you guys at REW aren't too worried about it, i won't be. Thanks for the information.