Why do I have references to robots txt in my web stat programs?

Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.

  • 524 Users Found This Useful

Was this answer helpful?

Related Articles

Enable Mod Rewrite

Create an .htaccess file with the following contents and upload it to your public_html directory....

Create a 410 Redirect for Missing Files

A redirect which is often forgot about is the 410 redirect. This is search engine friendly in the...

How do I hide my domain WHOIS information?

If you do not wish to have your contact information shown when someone does a WHOIS report on...

Disable error_log via htaccess

Prevent public display of PHP errors via htaccess# supress php errorsphp_flag...

Adding a Facebook Button

Facebook has a wonderful page on adding a Facebook image to your website complete with...