Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Why can users behind proxies not see my site?

If you have enabled hot link protection users behind proxies may not be able to see your site...

I can not get the transfer authorization email can it be forwarded to a different email address?

The authorization has to be authorized by the registered contact for the domain. We cannot change...

How do I create a helpdesk ticket?

You may create a helpdesk ticket at any time by visiting https://my.ultrawebhosting.com and...

What is this Code 304 appearing in my stats?

304 is typically sent as a header if a visitor re-requests a document and the document has not...

https:// with Weebly

To enable and force https:// with your Weebly website, be sure to go into your editor area,...