Robots.txt Generator
Build a valid robots.txt for your site: set your sitemap, block the paths you want private, and choose how AI crawlers like GPTBot and ClaudeBot may access your content. Copy or download the result.
Your robots.txt
The preview is editable — tweak it directly, then copy or download. Upload the file to your site root as /robots.txt.
How it works
Enter your site URL — the sitemap field auto-fills with the standard /sitemap.xml location (you can override it).
List any paths you want to keep crawlers out of, one per line, like /admin or /private.
Pick an AI-crawler policy: allow all, block all, or choose per bot from GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and others.
The file preview updates live and warns about problems like relative paths or a non-absolute sitemap URL. Edit it directly if you want.
Copy the result or download it, then upload it to your site’s root as /robots.txt.
Frequently asked questions
Where does the file need to go?
At the root of your domain: https://yoursite.com/robots.txt. Crawlers only look there — a robots.txt in a subfolder is ignored. Most hosts and CMSs let you place or override this file.
Should I block AI crawlers?
It is a trade-off. Blocking GPTBot, ClaudeBot, and similar bots keeps your content out of AI training and some AI answers — but it also means AI assistants cannot cite or recommend your site. For most businesses that want to be found, allowing AI crawlers is the better default.
What does Google-Extended control?
Google-Extended is not Google Search — blocking it opts your content out of training Google’s AI models (like Gemini) without affecting your search rankings. Blocking Googlebot itself, on the other hand, removes you from search.
Will Disallow rules hide my private pages?
No. robots.txt is a public file that politely asks well-behaved crawlers to stay out — anyone can read it, and it is not access control. Protect genuinely private content with authentication, and use noindex to keep pages out of search results.
Does this tool upload anything to my site?
No. The generator runs entirely in your browser and never contacts your server — it produces the file content, and you upload it yourself.
Get help fixing this
Fortitude builds fast, search-friendly websites. Leave your email and we'll help you fix what's holding your site back.
More free tools
SERP Snippet Preview
Preview how your title and meta description will look in Google search results.
Meta Description Checker
Check any page’s meta description for length, clarity, and click-worthiness.
H1 Checker
Find missing, duplicate, or weak H1 headings on any page.
Canonical Checker
Verify canonical tags are present, absolute, and pointing to the right URL.
Robots.txt Checker
Check your robots.txt for syntax issues and crawler access, including AI bots.
Sitemap Checker
Discover and validate your XML sitemap, including sitemap index files.
llms.txt Validator
Validate your llms.txt structure so AI assistants can understand your site.
llms.txt Generator
Generate a draft llms.txt from your site’s pages and description.
AI SEO Audit
A full AI-readiness SEO report: crawlability, metadata, structure, and fixes.