Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Lightweight Browser for Older and Slower Computers

Are you looking for a faster up to date browser in this current age for your older computer on...

Prevent CloudFlare from Loading a js or Script

Sometimes CloudFlare / Rocket Loader can have problems with a script and hose it up. To have...

Do you have a firewall?

Yes, our servers have firewalls. For security we close off unused ports and update the server...

vBulletin fsockopen() [function.fsockopen]: unable to connect

When setting up mail in vBulletin be sure you are using localhost for the mail server along with...

Creating and Submitting a Google Sitemap

Creating a Google sitemap can be easy and beneficial. By using one, you are pushing a full list...