🤖
robots.txt Generator
Runs in your browserConfiguration
Output
Ready — zero server calls
Related Tools
How to Use
- Enter your site URL and sitemap URL.
- Add disallowed paths and configure AI bot access.
- Copy the generated robots.txt to your site root.
Generate robots.txt
The robots.txt file tells search engines and web crawlers which pages to crawl or skip. Every website should have one at the root (example.com/robots.txt) to guide crawler behavior.
AI Bot Controls
With the rise of AI crawlers (GPTBot, ClaudeBot, Google-Extended, etc.), controlling which bots can access your content is more important than ever. This tool includes toggles for common AI bots.