Why Do I See robots.txt in My Web Stats

If you see "robots.txt" appearing frequently in your AWStats or Webalizer reports, that's normal. Search engine bots (Googlebot, Bingbot, etc.) check your robots.txt file before crawling your site. Every visit from a bot generates a hit on robots.txt. This is expected behavior and not a problem.

01. What Is robots.txt

Robots.txt is a text file in your website's root directory (public_html/robots.txt) that tells search engine bots which pages they should and shouldn't crawl. Before a bot crawls any page on your site, it checks robots.txt first.

If you don't have a robots.txt file, bots will look for it anyway (getting a 404), and you'll still see it in your stats. Creating one gives you control over what gets indexed.

02. Creating a Basic robots.txt

Create a file called robots.txt in your public_html directory with this content:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

This tells all bots they can crawl everything, and points them to your sitemap. WordPress generates a sitemap automatically (usually at /wp-sitemap.xml), or you can use a plugin like Yoast SEO to create one.

For blocking bots from specific directories or for blocking abusive crawlers, see our Resource Limit guide which covers bot blocking techniques.

03. Understanding Your Web Stats

cPanel includes two web statistics tools in the Metrics section:

• AWStats - detailed traffic reports with visitor counts, bandwidth, referrers, and search engine activity
• Webalizer - simpler visual reports with monthly traffic graphs

Both tools count robots.txt hits. This inflates your total "hits" count but does not affect your actual visitor count or bandwidth in any meaningful way. For more accurate visitor analytics, use Google Analytics (free), which only tracks real human visitors.

Last updated March 2026 · Browse all SEO articles