Robots.txt Generator

Allowing AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and similar) makes it more likely assistants and AI search products can fetch, cite, and recommend your pages. Blocking them can limit training and some answer-engine use of your content, but it also reduces AI discovery and citations. For most marketing and content sites that want visibility, allow AI bots by default and only block specific user-agents if you have a clear licensing or brand reason.

Googlebot is Google Search’s crawler — keeping it allowed is essential for organic rankings and indexation. Google-Extended is a separate token for Google’s generative AI / training-related use; blocking Google-Extended does not remove you from Google Search. When deciding rules, allow Googlebot for SEO, and treat Google-Extended as an independent choice about AI-model use of your content.

No. Disallow only asks well-behaved crawlers not to fetch those URLs; robots.txt is public, so the paths you list are visible to anyone. It is not authentication, encryption, or access control. Use login, firewall, or server rules for real secrets, and noindex (or similar) when your goal is keeping a page out of search results rather than merely discouraging crawls.

Start with a permissive User-agent: * that allows crawlable marketing content, keep Googlebot and Bingbot unrestricted, and allow major AI crawlers rather than blanketing them with Disallow. Add a Sitemap: line pointing at your absolute XML sitemap URL so search and AI bots can discover URLs efficiently. Reserve Disallow for admin, cart, account, internal search, and staging paths — not for pages you hope AI answers will quote.

Common safe Disallow targets include /admin, /wp-admin, /cart, /checkout, /account, /login, /search, and preview or draft URL prefixes — thin or transactional URLs that waste crawl budget and rarely belong in results. Always prefer absolute Sitemap: URLs (https://example.com/sitemap.xml), and you may list more than one if you have separate sitemaps. Do not Disallow the same URLs you need indexed or cited; sitemap discovery only helps for URLs that are still allowed to be crawled.