Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Sub-Domain times out

When you add a sub-domain and it times out make sure you do not have redirection set from that...

How to create a temporary 302 redirect via htaccess

An easy way to create a temporary redirect from one page to another web page is to use the 302...

SSL - Creating a CSR in Windows 2003

Creating the CSR IIS Windows 2003 or 2000 Server: From Administrative Tools, run the Internet...

What is a domain?

A domain is a human-readable name assigned to an IP address to make accessing websites much...

Can I change my username?

Usernames are setup in the manner for security. Most programs will allow you to save the...