Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Unable to view Ultrawebhosting.com from my foreign browser

We are an American company based out of Seattle, WA. Our website is written in English as a...

Do you support TeamSpeak?

We do not support Teamspeak on our servers but allow you to create a subdomain such as...

Why is the server load status red?

The server load status will be red if the server is under a little more stress than usual. This...

What is a domain?

A domain is a human-readable name assigned to an IP address to make accessing websites much...

Creating a Form Button

Sometimes all you need is a simple button. The following pages are pretty useful tools for...