Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Domain Slamming - Beware

There are several companies out there using a predatory practice of attempting to have you renew...

How do I cancel my account?

We are sorry to hear you would like to cancel your account! If there is anything we can do,...

Redirect Internet Explorer visitors to a supported browsers page

Redirect users of Internet Explorer to a supported browsers page or any other page.  To do this,...

Error 406 unacceptable

This error can be caused by several factors. One cause can be the incorrect permissions are set...

Lightweight Browser for Older and Slower Computers

Are you looking for a faster up to date browser in this current age for your older computer on...