Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Why can I not ping my domain?

For security purposes we have closed off the ICMP port which is used for ping.

Why can users behind proxies not see my site?

If you have enabled hot link protection users behind proxies may not be able to see your site...

I need a module installed but I do not have rights

No problem. Send us a trouble ticket and we will install the module as long as there are no...

I am unable to delete a file

This can occur for several reasons. With our service, if a directory of a script was found to be...

I updated my site but I still see old pages

Make sure your files are indeed updated by checking the file time stamps. Also, you may want to...