Automated bots operated by AI companies to discover, retrieve, and index web content for training AI models and populating AI assistant responses. Major AI crawlers include GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google), and OAI-SearchBot. These crawlers respect robots.txt directives — you can block them if desired, though blocking may reduce your visibility in their AI systems. Understanding which crawlers access your site and configuring robots.txt accordingly is part of AI SEO strategy. Learn more in our AI crawlers guide.
Part of the Plain Intelligence glossary. See all terms.