Which AI crawlers exist, and how to let them in

Every AI platform reaches your site through named crawlers, and a surprising number of brands block them by accident, usually via a copy-pasted robots.txt or an overzealous bot-protection rule. If assistants can't read your site, the content you publish for them might as well not exist. Here's who's knocking, and a sane door policy.

Two kinds of AI bot: the distinction that matters

• Training crawlers collect content to train future models. Blocking them is a philosophical/licensing choice with slow, diffuse effects.
• Retrieval agents fetch your pages live, while answering a user, or build the search index the assistant queries. Blocking these has immediate, visible effects: the assistant can't cite you, and may not find you at all.

Most accidental self-harm comes from blocking the second kind while meaning to block the first.

The crawlers, by platform

OpenAI (ChatGPT)

• GPTBot: training crawler.
• OAI-SearchBot: builds ChatGPT's search index. Block it and you fall out of ChatGPT search results.
• ChatGPT-User: live fetches when a user's chat browses your page. Block it and ChatGPT can't read or cite you in-session.

Anthropic (Claude)

• ClaudeBot: training crawler.
• Claude-SearchBot: search indexing.
• Claude-User: live fetches on behalf of users.

Perplexity

• PerplexityBot: index crawler.
• Perplexity-User: live fetches for user queries.

Google

• Googlebot: the classic search crawler. AI Overviews and AI Mode are built on Google Search, so ordinary Google SEO access covers them.
• Google-Extended: a robots token (not a separate crawler) controlling use of your content for Gemini model training. Blocking it does not remove you from AI Overviews or Google Search.

Others worth knowing

• Bingbot: Bing's index feeds several assistants' web results.
• CCBot: Common Crawl; its archives feed many research and training datasets.
• Amazonbot, Applebot (+ Applebot-Extended training token), meta-externalagent, Bytespider: the long tail of platform crawlers, mostly training-oriented.

A sane default policy

For a commercial brand that wants AI visibility, the pragmatic default is: allow retrieval, decide separately on training.

# robots.txt: allow AI answer/retrieval bots
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Training crawlers: your call; blocking these does NOT
# remove you from AI answers that browse the live web.
# User-agent: GPTBot
# Disallow: /

Remember that robots.txt is only half the story: CDN and bot-protection rules (Cloudflare, WAFs, rate limiters) silently 403 many AI agents regardless of robots.txt. If a platform never cites you despite good content, check your firewall logs before your content strategy.

Verify instead of assuming

The only proof is your own traffic: check server or CDN logs for the user-agents above, and confirm they're getting 200s on the pages that matter. Geofound's crawler tracking does this continuously, and its free scan shows you the downstream effect: whether ChatGPT actually names you when your buyers ask.