This free robots.txt generator builds a valid, copy-ready robots.txt file from simple controls — allow or block specific crawlers, add disallow paths, declare your XML sitemap, and set explicit rules for AI crawlers like GPTBot, ClaudeBot, and PerplexityBot. Copy the result or download it as a ready-to-upload robots.txt file.
# Your generated robots.txt will appear here
How to use this robots.txt generator
- Set your default policy. Almost every site should allow everything and then list specific paths to block.
- Add disallow paths — admin areas, internal search results, cart and checkout pages, and parameter-heavy URLs are the usual candidates.
- Decide your AI crawler policy. Blocking training bots like GPTBot or Google-Extended keeps your content out of model training, but blocking search bots like OAI-SearchBot or PerplexityBot also removes you from AI search answers and citations — a real visibility trade-off.
- Add your sitemap URL so every crawler can find your XML sitemap without guessing.
- Generate, then upload the file to your site root as
/robots.txt— it must live at the root to work.
Why robots.txt matters more in the AI search era
Robots.txt has always been the first file a crawler requests, and our robots.txt guide covers the classic crawl-control use cases. What changed is who is crawling: AI systems now send distinct user agents for training and for live search, and treating them identically is a strategy mistake. Understanding how AI crawlers work and pairing robots.txt with an llms.txt file gives you granular control over both crawl access and AI visibility. If crawl efficiency is your concern, start with crawl budget optimization and our crawlability guide.
Related tools and guides
- XML Sitemap Generator — create the sitemap your robots.txt should declare.
- SEO Meta Tag Generator — page-level robots directives live in meta tags.
- Schema Markup Generator — structured data for the pages you want crawled.
- Technical SEO checklist — robots.txt in the context of a full audit.
Frequently Asked Questions
Where does the robots.txt file go?
At the root of your domain: https://example.com/robots.txt. Crawlers only look there — a robots.txt in a subdirectory is ignored. Each subdomain needs its own file, so blog.example.com and example.com are controlled separately.
Does robots.txt stop a page from being indexed?
Not reliably. Robots.txt controls crawling, not indexing — a blocked URL can still appear in results if other sites link to it, just without a description. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header instead.
Should I block AI crawlers like GPTBot?
It depends on your goal. Blocking training bots (GPTBot, Google-Extended, CCBot) keeps content out of future model training. But blocking retrieval bots (OAI-SearchBot, PerplexityBot, ClaudeBot) removes you from AI search answers and citations — which is where a growing share of discovery happens. Many publishers now allow retrieval while blocking training.
What is the difference between Disallow and noindex?
Disallow (robots.txt) tells crawlers not to fetch a URL. Noindex (meta tag) tells search engines not to show a fetched page in results. They are complementary: use Disallow to save crawl budget on unimportant sections, and noindex to keep crawlable pages out of search results.
Can a robots.txt mistake take my site out of Google?
Yes — a single Disallow: / under User-agent: * blocks your entire site from crawling, and rankings decay soon after. That is why this generator makes the block-everything option explicit rather than a default, and why you should always test changes in Google Search Console's robots.txt report.
Not sure what your crawl setup should look like? Our technical SEO service includes crawl-control strategy, or browse all free tools.