Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 522 Users Found This Useful

Was this answer helpful?

Related Articles

How do I submit my site to search engines?

Submit your website to popular search engines to increase your webite's exposure and visitor...

Adding HTML to Site Builder

When adding HTML code/source in site builders be sure you are in HTML mode. There is often a...

View your website in older browser versions

As developers and webmasters it is always important to make sure your website works in several...

Unable to view Ultrawebhosting.com from my foreign browser

We are an American company based out of Seattle, WA. Our website is written in English as a...

I can not get the transfer authorization email can it be forwarded to a different email address?

The authorization has to be authorized by the registered contact for the domain. We cannot change...