Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Inode issues

There are many issues that can arise from reaching your inode limit. The inode limit is the total...

How do I parse html files as shtml?

Create an .htaccess file in the directory needed and add the following: AddHandler...

Domain Slamming - Beware

There are several companies out there using a predatory practice of attempting to have you renew...

Password Protect Directories

Within control panel select Password Protection. Select the directory you would like to protect....

I can not get the transfer authorization email can it be forwarded to a different email address?

The authorization has to be authorized by the registered contact for the domain. We cannot change...