AI crawler
Also known as: AI bot, GPTBot, ClaudeBot
An AI crawler is an automated program that reads websites on behalf of an AI system, for search indexing, live user fetches, or model training.
In full
An AI crawler is an automated program operated by an AI company to read web pages. Crucially, they do three different jobs and the distinction determines whether blocking one costs you anything. Search-indexing crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot build the index an assistant retrieves from when answering; blocking these makes citation impossible. User-triggered fetchers such as ChatGPT-User and Perplexity-User read a specific page because a person asked about it. Training crawlers such as GPTBot and ClaudeBot collect content that may inform a future model, and blocking them has no effect on current citations. Each identifies itself with a distinct user-agent string, and several operators publish IP ranges so the bot can be verified.
Why it matters
Many websites have blocked the crawler with no citation impact while leaving the one that matters untouched, or have blocked all of them accidentally through a security product, making themselves invisible to AI search without realising.
A common misconception
That Google-Extended controls whether you appear in Google's AI Overviews. It does not. AI Overviews are served from the ordinary Googlebot index; Google-Extended governs Gemini training and grounding only.