AI Crawlers and Access Control

Track 2 · Lesson 2 of 6 · about 15 minutes

AI crawlers and access control

You cannot make an informed decision about AI access until you know who is knocking and what each visitor is actually for. The names matter, because blocking the wrong one has opposite consequences to the ones you intended.

Two very different reasons to fetch your page

Training — collecting text to improve future models. Your content becomes part of what the model learns from; there is no citation and no traffic.

Retrieval — fetching pages to answer a live question, as covered in lesson one. This is the path that produces citations and clicks.

Most major providers now split these across separate user agents, which is what makes a deliberate policy possible.

The tokens worth knowing

GPTBot is OpenAI’s training crawler; OAI-SearchBot feeds ChatGPT’s search results; ChatGPT-User fetches when a user’s request needs a live page. Blocking GPTBot opts you out of training while leaving search citation intact; blocking OAI-SearchBot removes you from the surface where you might have been cited.

ClaudeBot crawls for Anthropic; PerplexityBot and Perplexity-User serve Perplexity’s index and live fetches. Google-Extended is the odd one out: it is not a crawler you will see in logs but a control token that opts your content out of Gemini training and grounding. It does not affect Google Search rankings — and, importantly, it is not a switch for AI Overviews, which draw on the ordinary Search index via ordinary Googlebot.

CCBot (Common Crawl) sits upstream of many training datasets: block it and you quietly exit several pipelines at once.

What robots.txt can and cannot do

All of this runs through robots.txt user-agent groups, and all of it is voluntary. The major providers publish their tokens and state that they comply; nothing technically enforces it. robots.txt is a fence made of convention — real enough that the big actors respect it, useless against anyone who chooses not to.

The trade is yours to weigh. An editorial site may happily feed retrieval while opting out of training. A site whose entire value is being cited should think hard before blocking any retrieval agent. The only wrong version is the accidental one — a blanket block copied from a blog post, silently removing you from surfaces you wanted.

And llms.txt?

A community proposal for a file listing your key content for language models. Publishing one is cheap and harmless; no major provider has confirmed using it. It belongs in the “experiment” column, not the “control” column — the controls that demonstrably work are the user-agent groups above.

Going deeper: audit your own AI crawler access · llms.txt vs robots.txt vs sitemap.