Robots.txt Checker

No. A Disallow rule only asks crawlers not to fetch a URL; it does not remove that URL from the index. If other sites link to a blocked page, Google can still list it—often with a sparse or missing snippet. To keep a page out of search results, use a noindex meta tag or HTTP header (on a crawlable page), password protection, or removal tools in Search Console.

Major AI crawlers generally honor robots.txt the same way search bots do: User-agent groups and Allow/Disallow patterns control which paths they may fetch. You can allow Googlebot while blocking GPTBot, ClaudeBot, or PerplexityBot (or the reverse) with separate user-agent blocks. Blocking AI bots is a content-access choice—it does not hide pages from Google or Bing search by itself.

A Sitemap: line points crawlers to your XML sitemap URL so they can discover pages without relying only on links or manual Search Console submits. It is a discovery hint, not a ranking boost, and it does not override Disallow rules for URLs listed in the sitemap. Absolute https URLs work best; you can list more than one sitemap if you use separate files or indexes.

No. Robots.txt is public—anyone can read which paths you Disallow, and Disallow only asks well-behaved bots not to crawl. It does not stop browsers, scrapers that ignore the file, or people who already know the URL. Protect admin and private areas with authentication, correct server permissions, and noindex where appropriate—not robots.txt alone.

Most crawlers treat a missing robots.txt as full crawl access: they may fetch any publicly reachable URL. That is fine for many small sites, but you lose a standard place to declare your sitemap and to set rules for Googlebot, Bingbot, or AI bots like GPTBot. A simple, valid file at https://yoursite.com/robots.txt is still recommended once you care about crawl guidance or bot-specific access.