 |
|
|
Thread Tools | Display Modes |
12-04-2006, 02:26 PM
|
||||
|
||||
what is this robots.txt
why is google asking for it, how do i install it. Can someone tell me where to get it
__________________
Looking for Real Estate, Investments, Condos in Dallas-Fort Worth-Denton-Keller-Haslet-North Texas Area. We have you covered, 400 New Homes and 60,000 Pre-owned home. Mike Pannell 817-703-3238 Dallas Real Estate | Fort Worth Texas Real Estate | Dallas Texas Real Estate BTW for all you COWBOY Haters. Superbowl Bound Baby. Sorry Deadskins 
|
|
12-04-2006, 02:53 PM
|
|||
|
|||
|
Re: what is this robots.txt
I believe it's a supplemental file that you include in your root folder that includes the pages that you do not want the spiders to include in in their crawl. I don't want any of my pages excluded so I don't use this file and am not sure how to format it. Someone please correct me if I'm wrong on this one.
__________________
Eric Rogers - REALTOR Century 21 Pro-Team Aurora IL Real Estate | Batavia Real Estate Yorkville Real Estate
|
|
12-04-2006, 03:29 PM
|
|||
|
|||
|
Re: what is this robots.txt
Yes you are right, you just create a file named robots.txt and the spider won't crawl those pages this is what it should look like
User-agent: * Disallow: /webstats/ (or whatever you don't want the world to see!)
__________________
Mortgage Orlando Specialist. Concentrating in Equity Repositioning and considering a mortgage as a financial planning tool.
|
|
12-04-2006, 03:33 PM
|
||||
|
||||
|
Re: what is this robots.txt
i am just wondering why google says i need it. It tells me i need it on the webmaster tools thing they have
__________________
Looking for Real Estate, Investments, Condos in Dallas-Fort Worth-Denton-Keller-Haslet-North Texas Area. We have you covered, 400 New Homes and 60,000 Pre-owned home. Mike Pannell 817-703-3238 Dallas Real Estate | Fort Worth Texas Real Estate | Dallas Texas Real Estate BTW for all you COWBOY Haters. Superbowl Bound Baby. Sorry Deadskins
|
|
12-04-2006, 10:03 PM
|
|||
|
|||
|
Re: what is this robots.txt
Google wants to know that it's actually allowed to crawl your web site before doing so. It doesn't trust you on your word... it wants to see that it is allowed to spider your site. Google may also ask you to post a "key" to your web site for enhanced functionality.
Robots.txt is well documented throughout the web.
|
|
«
Previous Thread
|
Next Thread
»
| Thread Tools | |
| Display Modes | |
Linear Mode
|
|
