AI crawlers, and which ones to let in

Most advice about AI crawlers treats them as one category to be allowed or blocked wholesale. That is the mistake that quietly costs businesses their AI visibility. OpenAI alone operates four distinct bots that do entirely different things, and the one that determines whether ChatGPT can cite you is not the one most robots.txt files block.

Three jobs hide behind one label

Every bot in this space is doing one of three things, and the distinction decides everything. Search indexing bots build the index an assistant retrieves from when it answers a question. If you block one of these, you cannot be cited by that assistant, no matter how good your content is. User-fetch bots go and read a specific page because a person asked the assistant about it; blocking these breaks the experience for someone who was already interested enough to paste your URL. Training bots collect content that may inform a future model. Blocking a training bot has no effect on whether you are cited today, which is why it is the one genuinely defensible thing to block if bandwidth is a concern.

The full crawler table

Recommendations below assume you want maximum visibility in AI answers. A publisher whose business model depends on page views may reasonably make different calls on the training column, and that is a legitimate commercial decision rather than a technical one.

Google-Extended does not do what its name suggests

This one catches almost everybody. It is natural to read Google-Extended as the switch that controls whether Google's AI features use your content, and to block it if you would rather not feed them. It is not that switch. Google's own crawler documentation states that Google-Extended governs whether content is used to train Gemini models and to ground certain Gemini and Vertex products, and that it does not affect a site's inclusion in Google Search and is not used as a ranking signal. AI Overviews and AI Mode are served from the ordinary Googlebot index. There is currently no mechanism to appear in Google Search while opting out of AI Overviews. Blocking Google-Extended costs you grounding in the Gemini app and gains you nothing in Search.

Robots.txt is a request, not a lock

Two things worth being clear-eyed about. First, robots.txt is advisory. Well-behaved operators honour it and several publish IP ranges so you can verify their bots, but a crawler that ignores it will keep ignoring it, so anything you genuinely need to stop has to be stopped at the network edge rather than in a text file. Second, OpenAI's own documentation notes that because ChatGPT-User actions are initiated by a person rather than by a crawl schedule, robots.txt rules may not apply to them. The practical implication is that robots.txt is the right tool for expressing a preference to cooperative operators and the wrong tool for enforcement.

What allowing everything actually costs

Bandwidth, mostly, and the ratio is worse than people expect. Cloudflare's published analysis of AI bot traffic found that the large majority of it is training collection rather than search indexing or user-triggered fetches, and its crawl-to-referral ratios show some operators crawling tens of thousands of pages for every visitor they send back. For a brochure or services site with modest page counts, this is a rounding error and the trade is clearly worth making. For a large publisher serving millions of pages, it is a real infrastructure cost and a real commercial decision. Know which you are before copying anyone's robots.txt.

How to check what you are currently doing

1. 01Open yoursite.com/robots.txt in a browser. If it returns nothing or a 404, you are allowing everything by default, which for most businesses is the correct posture anyway.
2. 02Look for Disallow lines under any of the user agents in the table above. A blanket Disallow under a named AI bot is the thing to check first.
3. 03Check your hosting or CDN settings separately. Several security products block unfamiliar user agents by default, which blocks AI crawlers without anything appearing in robots.txt at all. This is the most common cause of invisible blocking.
4. 04Check your server logs for the user agents above. If a bot has never requested anything, something upstream is stopping it.
5. 05Confirm the site is indexed in Bing, not just Google. Bing is the entire retrieval layer for Microsoft Copilot and contributes to ChatGPT, and plenty of sites are indexed in one and not the other.