Every AI assistant that cites the web depends on crawlers — and your robots.txt quietly decides which of them can see you. Most sites have never made that decision deliberately; they inherit defaults that either overexpose or, worse, silently block the AI crawlers that power ChatGPT, Claude, Perplexity, and Gemini answers. This guide explains what each major bot actually does, the trade-offs of blocking versus allowing, and the exact robots.txt patterns to implement your policy. It underpins every platform guide in our AI SEO series.
What AI Crawlers Do (Three Different Jobs)
AI crawlers are automated agents that fetch web content for one of three purposes: training models, building search indexes, or retrieving pages live during a user’s question. The same company usually runs separate bots for each job — and you can permit them independently.
| Job | Example bots | Blocking means |
|---|---|---|
| Model training | GPTBot, Google-Extended, CCBot | Your content stops teaching future models |
| Search indexing | OAI-SearchBot, PerplexityBot, Bingbot | You drop out of AI search retrieval pools |
| Live user fetches | ChatGPT-User, Perplexity-User, Claude-User | Assistants cannot open your pages mid-answer |
Confusing these jobs causes the classic mistake: blocking a training bot to protect content, and accidentally assuming that also governs search visibility. It does not — the jobs are separable, which is the entire point of this taxonomy. The pipeline they feed is mapped in How AI Search Works Behind the Scenes.
The Bot Roster: Who Runs What
- OpenAI — GPTBot (training), OAI-SearchBot (ChatGPT search index), ChatGPT-User (live fetches). Documented at platform.openai.com/docs/bots. Directly affects ChatGPT visibility.
- Anthropic — ClaudeBot (crawl) and user-triggered fetchers; governs Claude citations.
- Perplexity — PerplexityBot (index) and Perplexity-User (live); documented at docs.perplexity.ai; the backbone of Perplexity SEO.
- Google — Googlebot (Search + AI Overviews + AI Mode) and Google-Extended (Gemini training/grounding toggle); the nuance that trips everyone is unpacked in Gemini Search Optimization.
- Microsoft — Bingbot feeds Bing, Copilot, and ChatGPT search retrieval; see Copilot optimization.
- Common Crawl — CCBot builds the open dataset many models train on; blocking it trims broad training exposure with no search cost.
robots.txt Patterns for Three Policies
Full visibility (most businesses): allow everything above — the default file needs no AI-specific lines at all.
Search visibility without training (common for publishers):
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Search and live-fetch bots stay allowed; training bots are refused.
Full AI opt-out: extend the disallow list to OAI-SearchBot, PerplexityBot, ClaudeBot and the user-fetch agents — accepting near-total AI invisibility. Whichever policy you pick, implement it in a clean robots.txt, and remember crawlers must be able to read the file itself — a misconfigured crawl setup defeats every directive.
Verifying What Actually Crawls You
Policies are theory; server logs are truth. Check access logs for the user-agent strings above (our log file analysis guide shows how), confirm robots.txt returns 200 for each bot, and watch for security layers silently 403-ing legitimate crawlers. Then verify outcomes at the answer layer: run monthly citation spot-checks per platform using the workflow in AI Search Analytics. Strategy-level guidance for what to expose lives in our AI SEO guide.
- AI bots do three separable jobs — training, search indexing, live fetching — and can be permitted independently.
- Blocking GPTBot or Google-Extended does not remove you from AI search; blocking index/fetch bots does.
- Every major operator documents its bots publicly; robots.txt directives are honored by the reputable ones.
- Choose a deliberate policy (full visibility, search-without-training, or full opt-out) and implement exactly that.
- Trust server logs over assumptions — security stacks frequently block legitimate AI crawlers by accident.
Frequently Asked Questions
Do AI crawlers actually respect robots.txt?
The major operators — OpenAI, Anthropic, Google, Microsoft, Perplexity, Common Crawl — publicly commit to honoring robots.txt and publish their user-agents. Disreputable scrapers exist and ignore everything, but the bots that matter for AI search visibility follow the rules, which is what makes deliberate policy worthwhile.
Will blocking training bots protect my content from AI?
It stops future model training on your pages by compliant operators, but content already in past training snapshots stays there, and search-time retrieval can still read allowed pages. Think of it as reducing future training exposure, not retroactive removal.
Does blocking AI crawlers affect my Google rankings?
No — with one caveat. GPTBot, ClaudeBot, PerplexityBot, CCBot, and Google-Extended have no effect on Google Search rankings. Only blocking Googlebot itself (or Bingbot for Bing) changes classic search presence. Training opt-outs and ranking are fully independent systems.
How do I block AI training but stay in AI search results?
Disallow GPTBot, Google-Extended, and CCBot while leaving OAI-SearchBot, PerplexityBot, ClaudeBot, Bingbot, and the user-fetch agents allowed. This keeps you retrievable and citable in ChatGPT search, Perplexity, Copilot, and Gemini while refusing training use by compliant operators.
Should I add a crawl-delay for AI bots?
Rarely necessary — AI crawlers are modest compared with search engines, and aggressive delays risk incomplete indexing. If server load genuinely spikes, prefer rate-limiting at the CDN with verified bot allowances over blunt robots.txt delays that some bots ignore anyway.
Conclusion
Your robots.txt is now an AI visibility policy document. Decide what you want — training exposure, search citations, both, or neither — write the four lines that implement it, and verify in your logs. Deliberate beats default every time. Next, see how the systems that read your pages decide who they are: AI Knowledge Graphs Explained.
See how your site actually shows up in AI search. An AI visibility audit maps where you’re cited, where you’re invisible, and what to fix first — in plain English.
Get your AI visibility auditTry the free SEO tools →
Prefer self-serve? The interactive checklists turn guides like this one into a working to-do list.
Keep reading in AI SEO
Get one email when something genuinely changes
AI search moves fast and most of it is noise. We send one short email when a real shift is worth your time. Unsubscribe anytime.
Published by Plain Intelligence — practical AI SEO, GEO, and technical SEO, documented in plain English. About Plain Intelligence →
↑ Back to AI SEO · Explore all articles · Free tools & resources · Glossary