This free AI crawler access auditor reads your robots.txt and shows instantly which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 16 more — you’re allowing or blocking. If the bots behind ChatGPT, Claude, Perplexity, and Gemini can’t reach your site, you can’t be cited in their answers. Paste your robots.txt below; nothing leaves your browser.
How to use the AI crawler access auditor
- Open your own
robots.txtathttps://yoursite.com/robots.txtand copy its contents. - Paste them into the box above and click Audit access.
- Review the table: each of 20 AI crawlers is shown as Allowed or Blocked, with which rule decided it.
- If a bot you want citing you is blocked by accident, fix it with the robots.txt generator.
- Re-paste and re-audit to confirm the change.
Why AI crawler access decides your AI visibility
Before any AI engine can cite you, its crawler has to be allowed to read your pages. Many sites block AI bots by accident — through a copied robots.txt, a security plugin, or a default “block scrapers” setting — and then wonder why they never appear in AI answers. Each engine uses its own user-agents: OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot, PerplexityBot, and Google’s Google-Extended. This auditor checks all of them at once so you can make blocking a deliberate choice, not an accident. It is the access side of generative engine optimization; for the full picture see AI Search Explained and our crawlability guide.
Related tools and guides
- Robots.txt Generator — build a correct robots.txt with per-bot AI controls.
- llms.txt Generator — point AI crawlers at your best pages once they can reach you.
- AEO Readiness Analyzer — score how citable each page is.
- Crawlability Guide — the fundamentals of getting crawled and indexed.
- AI Citation Tracker — measure whether AI engines actually cite you.
Frequently Asked Questions
How does the tool decide if a bot is blocked?
For each AI crawler it looks for a matching User-agent group in your robots.txt. If one exists, those rules apply; if not, the bot falls back to the User-agent: * rules. A bot is reported as “Blocked” when the most specific matching rule disallows the site root (Disallow: /). Everything runs in your browser — nothing is uploaded.
Should I block AI crawlers?
Only if you have deliberately decided your content should not appear in AI answers. Blocking removes you from consideration entirely. For most sites seeking visibility, the pragmatic choice is to allow reputable AI crawlers while monitoring how they use your content. Blocking a training bot but allowing a search bot is a valid middle ground for some publishers.
Does “Allowed” guarantee I’ll be cited?
No. Access is necessary but not sufficient. Being cited also depends on your content being well-structured and extractable, your topical authority, and freshness. Think of an “Allowed” result as clearing the first gate — the crawler can reach you — after which structure and authority decide whether you are actually quoted.
What’s the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI’s crawler that indexes content and can be used for training and search; ChatGPT-User and OAI-SearchBot handle user-triggered browsing and search results. Because they are separate user-agents, you can allow or block them independently — which is why the auditor lists each one separately.
Is this auditor free and private?
Yes — free, no account, no limits, and it runs entirely in your browser, so the robots.txt you paste is never sent anywhere. It is part of the free Plain Intelligence resource hub.
Want your crawl and indexing foundations audited properly? See our technical SEO service, or browse all free tools in the resource hub.