Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Do you support TeamSpeak?

We do not support Teamspeak on our servers but allow you to create a subdomain such as...

How do I customize error messages for my site?

Within control panel select Custom Error Pages. Select the error code link and place whatever...

Do you have a firewall?

Yes, our servers have firewalls. For security we close off unused ports and update the server...

Inode issues

There are many issues that can arise from reaching your inode limit. The inode limit is the total...

My pages are updating/refreshing slowly

Refresh is immediate on the server. When a file is updated and uploaded it is there. If you are...