Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

I updated my site but I still see old pages

Make sure your files are indeed updated by checking the file time stamps. Also, you may want to...

What are all these htaccess files?

.htaccess files are used to perform certain rules such as displaying a 404.shtml file in replace...

Can I change my username?

Usernames are setup in the manner for security. Most programs will allow you to save the...

403 Error on POST

This can occur for several reasons: Be sure your file permissions are correct. If the file needs...

Error 406 unacceptable

This error can be caused by several factors. One cause can be the incorrect permissions are set...



Save 30% on web hosting - Use coupon code Hosting30