Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Lightweight Browser for Older and Slower Computers

Are you looking for a faster up to date browser in this current age for your older computer on...

Error 508 / 503 - Resource Limit Reached

To help maintain the stability of servers and keep websites fast, UltraWebHosting has resource...

My pages are updating/refreshing slowly

Refresh is immediate on the server. When a file is updated and uploaded it is there. If you are...

html or htm parsed as shtml not working

To parse .html or .htm files as .shtml add the following to your .htaccess file in the...

Why does it show the Ultra Web Hosting home page when I go to my site?

There are several reasons for this. Ensure you do not have https at the beginning of your...