Many search engines will look for a robots.txt file before spidering your sites content. The robots.txt file contains specific rules a robot/spider should abide by when capturing data.
- 524 Users Found This Useful
Related Articles
Why can I not ping my domain?
For security purposes we have closed off the ICMP port which is used for ping.
Why can users behind proxies not see my site?
If you have enabled hot link protection users behind proxies may not be able to see your site...
I need a module installed but I do not have rights
No problem. Send us a trouble ticket and we will install the module as long as there are no...
I am unable to delete a file
This can occur for several reasons. With our service, if a directory of a script was found to be...
I updated my site but I still see old pages
Make sure your files are indeed updated by checking the file time stamps. Also, you may want to...
