Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.
- 524 Users Found This Useful
Related Articles
Enable Mod Rewrite
Create an .htaccess file with the following contents and upload it to your public_html directory....
Create a 410 Redirect for Missing Files
A redirect which is often forgot about is the 410 redirect. This is search engine friendly in the...
How do I hide my domain WHOIS information?
If you do not wish to have your contact information shown when someone does a WHOIS report on...
Disable error_log via htaccess
Prevent public display of PHP errors via htaccess# supress php errorsphp_flag...
Adding a Facebook Button
Facebook has a wonderful page on adding a Facebook image to your website complete with...
