Robots.txt Checker

Blocking Googlebot or Bingbot almost always means lost search traffic, so it fails the check. Blocking AI crawlers like GPTBot or ClaudeBot can be a deliberate content-licensing decision, so the tool warns you it is happening and lets you judge whether it is intentional.

No — it only stops crawling. A page blocked in robots.txt can still appear in results (without a description) if other sites link to it. To keep a page out of the index, use a noindex meta tag or authentication instead.

It applies the same rules crawlers do: the most specific user-agent group wins, then the longest matching Allow/Disallow pattern wins, with wildcards (*) and end anchors ($) supported per RFC 9309. On a tie, Allow wins, matching Google’s documented behavior.

Not necessarily — no robots.txt means everything may be crawled, which is fine for many sites. You lose the ability to declare your sitemap there and to steer specific bots, which is why having a simple, valid file is still recommended.

A “Sitemap:” line in robots.txt tells any crawler where your XML sitemap lives without you registering it anywhere. It is the one robots.txt directive that helps discovery rather than restricting it, so the tool flags its absence.