Back to free tools
🤖
SEO
Robots.txt Generator
Build a valid robots.txt with a few clicks — including AI crawler blocking.
Options
robots.txt
User-agent: *
Allow: /
Disallow: /admin
Disallow: /private
Disallow: /cart
Sitemap: https://example.com/sitemap.xml
About this tool
Toggle options to allow or block crawlers, add disallow paths, set a sitemap URL, and optionally block major AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended).
What it does
- •Toggle allow/disallow
- •Block AI training crawlers
- •Sitemap directive
- •Multiple disallow paths
Use cases
- ★Launching a new site
- ★Blocking AI scrapers
- ★Hiding admin routes
FAQ
Where do I put robots.txt?
At the root of your site — for example https://example.com/robots.txt. It must be plain text.
Which AI crawlers should I block?
The common ones are GPTBot, ClaudeBot, CCBot, Google-Extended, and PerplexityBot. This tool blocks all five with one toggle.