Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Whitelisting Our Support System Email Address

Whitelist Instructions Because we use often use html emails with http addresses in them, our...

Default Nameservers

If you have signed up for an Ultra 1x, Ultra Unlimited, or Ultra Unlimited Pro plan the following...

What perl modules are installed on the server?

You can get a complete list of the modules that are installed by going to the control panel and...

Error 404 - File Not Found

Files are case sensitive. Make sure you are typing the address in correctly. Verify that the...

Why can I not ping my domain?

For security purposes we have closed off the ICMP port which is used for ping.