If you see "robots.txt" appearing frequently in your AWStats or Webalizer reports, that's normal. Search engine bots (Googlebot, Bingbot, etc.) check your robots.txt file before crawling your site. Every visit from a bot generates a hit on robots.txt. This is expected behavior and not a problem.
This Is Normal
Robots.txt hits in your stats come from search engine crawlers checking your crawl rules. This is healthy - it means search engines are indexing your site.
- Every search engine bot checks robots.txt before crawling
- High robots.txt hits = active crawling = good for SEO
- Not consuming meaningful bandwidth or resources
01. What Is robots.txt
Robots.txt is a text file in your website's root directory (public_html/robots.txt) that tells search engine bots which pages they should and shouldn't crawl. Before a bot crawls any page on your site, it checks robots.txt first.
If you don't have a robots.txt file, bots will look for it anyway (getting a 404), and you'll still see it in your stats. Creating one gives you control over what gets indexed.
02. Creating a Basic robots.txt
Create a file called robots.txt in your public_html directory with this content:
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlThis tells all bots they can crawl everything, and points them to your sitemap. WordPress generates a sitemap automatically (usually at /wp-sitemap.xml), or you can use a plugin like Yoast SEO to create one.
For blocking bots from specific directories or for blocking abusive crawlers, see our Resource Limit guide which covers bot blocking techniques.
03. Understanding Your Web Stats
cPanel includes two web statistics tools in the Metrics section:
- AWStats - detailed traffic reports with visitor counts, bandwidth, referrers, and search engine activity
- Webalizer - simpler visual reports with monthly traffic graphs
Both tools count robots.txt hits. This inflates your total "hits" count but does not affect your actual visitor count or bandwidth in any meaningful way. For more accurate visitor analytics, use Google Analytics (free), which only tracks real human visitors.
Status Codes in Your Stats (200, 304, 404)
Your stats and raw access logs show an HTTP status code next to each request. The common ones are normal, not errors:
- 200 OK - the file was served successfully.
- 304 Not Modified - the visitor's browser already had the file cached, and the server confirmed the cached copy is still current, so nothing was re-sent. A 304 saves bandwidth and is a good sign, not an error.
- 404 Not Found - a requested file does not exist. Usually a bot probing for files, or a broken link on your site.
Concerned About Bot Traffic?
If bots are consuming excessive resources (not just showing up in stats), see our resource limit guide for blocking techniques, or open a ticket.
Open a Support TicketLast updated March 2026 · Browse all SEO articles
