Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.
- 524 Users Found This Useful
Related Articles
Enable Mod Rewrite
Create an .htaccess file with the following contents and upload it to your public_html directory....
Can you write a script for me?
We do not write scripts but will be more than happy to guide you in the right direction. Most the...
I updated my site but I still see old pages
Make sure your files are indeed updated by checking the file time stamps. Also, you may want to...
Domain Slamming - Beware
There are several companies out there using a predatory practice of attempting to have you renew...
Meta Redirect
Adding this code to your web page will redirect your visitors to any address that you have...
