Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.
- 524 Users Found This Useful
Related Articles
Inode issues
There are many issues that can arise from reaching your inode limit. The inode limit is the total...
I updated my site but I still see old pages
Make sure your files are indeed updated by checking the file time stamps. Also, you may want to...
How can I redirect http to https?
Create an .htaccess file with the following contents and upload it to your public_html directory....
Error 401 Unauthorized
This can occur when a web page requires authorization or a login to view the contents. If none...
Password Protect Directories
Within control panel select Password Protection. Select the directory you would like to protect....
