Field guide / By CiteVisor Editorial / Updated 2026-11-10

AI crawler list: user agents, purposes and robots.txt rules

Most AI companies run more than one bot: one feeds search answers, one collects training data, and one fetches a page when a person asks. Treat them separately and you can stay visible in AI search without offering your content for model training.

Three kinds of AI crawler

It helps to sort AI bots by job before deciding what to allow.

  • Search crawlers build the index that answer engines quote and link to. Examples: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude, PerplexityBot for Perplexity, plus Googlebot and Bingbot for Google and Microsoft surfaces.
  • Training crawlers and controls decide whether your content may be used to train foundation models. Examples: GPTBot, ClaudeBot, CCBot, and the control tokens Google-Extended and Applebot-Extended.
  • User-initiated fetchers visit a page because someone asked a question inside the product. Examples: ChatGPT-User, Claude-User and Perplexity-User.

If your goal is to appear in AI answers, the first group matters most. Blocking a training crawler is a legitimate content decision, and on its own it does not remove you from the corresponding search product.

The reference table

Tokens are the names you use in robots.txt. Descriptions follow each operator's own documentation, reviewed on 2026-10-23.

Operator and tokenCategoryWhat the operator says it is forrobots.txt
OpenAI OAI-SearchBotSearchSurfacing websites in ChatGPT's search featuresRespected
OpenAI GPTBotTrainingContent that may be used to train OpenAI's foundation models (what GPTBot is)Respected
OpenAI ChatGPT-UserUser fetchCertain user actions in ChatGPT and custom GPTs (compared with OAI-SearchBot)May not apply
OpenAI OAI-AdsBotAds reviewChecking the safety of pages submitted as ChatGPT adsRespected, per OpenAI's advertiser guidance (October 2026)
Anthropic Claude-SearchBotSearchImproving the relevance and accuracy of Claude's search resultsRespected
Anthropic ClaudeBotTrainingContent that could contribute to model trainingRespected, with Crawl-delay
Anthropic Claude-UserUser fetchFetching pages when a person asks Claude a questionRespected
Perplexity PerplexityBotSearchSurfacing and linking websites in Perplexity; not used for foundation modelsRespected
Perplexity Perplexity-UserUser fetchVisiting a page to answer a user's questionGenerally ignored
Google GooglebotSearchGoogle Search, including eligibility for AI Overviews and AI ModeRespected
Google Google-ExtendedTraining controlA token that controls Gemini training and grounding use; it does not crawlToken only
Apple ApplebotSearchSearch in Spotlight, Siri and Safari; its data may also train Apple's foundation models and add context to AI answers in Siri and SearchRespected
Apple Applebot-ExtendedTraining controlOpting out of training Apple's foundation models; it does not crawl, and pages that disallow it can still appear in searchToken only
Common Crawl CCBotOpen datasetThe public Common Crawl corpus, which many model builders useRespected

Microsoft's Bingbot indexes the web for Bing. Bing Webmaster Tools now reports when indexed pages are cited in Microsoft Copilot and Bing's AI-generated answers, so Bing access matters for AI visibility too.

Apple's Applebot page, published 2026-09-04, adds a detail that matters for AI answers. Applebot data may give Apple's AI models context when they answer broad world knowledge questions in Siri and Search, and those answers may link to sources. Disallowing Applebot-Extended covers training only; to keep specific content out of those answers, Apple points to the nosnippet meta tag.

A robots.txt for "visible in AI search, no training"

The example below allows the AI search crawlers, declines the training crawlers and control tokens, and leaves ordinary search engines alone. Replace the private path and the sitemap URL with your own.

# AI search and answer engines
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/

# Model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Which bots each rule applies to

Each line of the example above reaches only the crawlers named in its group. Rule syntax and tokens were checked against OpenAI, Anthropic, Perplexity and Google documentation on 2026-11-10.

Rule in the exampleApplies toEffectOperator note (checked 2026-11-10)
User-agent: OAI-SearchBot + Allow: /OpenAI OAI-SearchBotStays eligible for ChatGPT search answersOpenAI: opted-out sites are not shown in ChatGPT search answers
User-agent: Claude-SearchBot + Allow: /Anthropic Claude-SearchBotStays indexable for Claude searchAnthropic: blocking it prevents indexing for search
User-agent: PerplexityBot + Allow: /Perplexity PerplexityBotStays visible in Perplexity resultsPerplexity: not used to crawl for foundation models
User-agent: GPTBot + Disallow: /OpenAI GPTBotOpts out of OpenAI model trainingOpenAI: disallowing signals content should not be used for training
User-agent: ClaudeBot + Disallow: /Anthropic ClaudeBotOpts out of Anthropic training collectionAnthropic: honors robots.txt directives
User-agent: Google-Extended + Disallow: /Google token only, no crawler of its ownOpts out of Gemini training and grounding useGoogle: no effect on Search inclusion or ranking
User-agent: *Every crawler without its own group, including Googlebot and BingbotNormal crawling except /admin/Applies only to bots with no named group

Not covered by any line: ChatGPT-User and Perplexity-User act when a person asks for a page, so robots.txt may not apply to them. Claude-User is a user-initiated fetcher too; Anthropic's page does not describe robots.txt handling for it separately.

OpenAI and Perplexity publish example user-agent strings. Version numbers change, so match on the token, not the whole string.

TokenExample user-agent string from the operator
OAI-SearchBotMozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
GPTBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
ChatGPT-UserMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PerplexityBotMozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
Perplexity-UserMozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Anthropic's help page names the ClaudeBot, Claude-User and Claude-SearchBot tokens but gives no full user-agent string, so none is listed here.

Want to see how a file like this reads for each crawler? The AI crawler robots.txt rules generator builds rules for 11 crawlers and shows a worked example of how this exact file evaluates for every one of them.

One detail causes many mistakes: a crawler follows only the most specific group that names it. Once you give OAI-SearchBot its own group, the rules under User-agent: * no longer apply to it. Repeat every private-path Disallow inside each named group, as the example does for /admin/.

Common mistakes

  • Blocking the wrong OpenAI bot. Adding GPTBot to a block list keeps content out of training. It does not hide the site from ChatGPT search, and blocking OAI-SearchBot to stop training removes you from search answers instead.
  • Expecting Google-Extended to control AI Overviews. Google says Google-Extended has no effect on Search inclusion or ranking. Search features, including AI Overviews, depend on Googlebot and the usual snippet controls such as nosnippet and max-snippet.
  • Forgetting the CDN. A crawler that robots.txt allows can still be stopped by a firewall rule, a bot challenge or an AI crawler block at your CDN. Check those settings separately; see our Cloudflare AI Crawl Control guide.
  • Treating robots.txt as access control. robots.txt states a preference. User-initiated fetchers may not follow it, and a misbehaving bot can ignore it. Content that must stay private needs authentication.
  • Expecting instant results. OpenAI says search can take about 24 hours to reflect a robots.txt change, and Perplexity says its systems may take up to 24 hours. The other operators here publish no timing.

How to check who is really visiting

User-agent strings are easy to fake, so verify the source IP before trusting a log line. OpenAI publishes IP lists for each bot, for example https://openai.com/searchbot.json and https://openai.com/gptbot.json. Perplexity publishes https://www.perplexity.com/perplexitybot.json, Anthropic publishes https://claude.com/crawling/bots.json, and Common Crawl publishes https://index.commoncrawl.org/ccbot.json.

Compare those lists with your server or CDN logs, and look for HTTP 403 responses or challenge pages served to search crawlers. If you use Cloudflare, AI Crawl Control lists the AI crawlers that visited, how often, and how many times each one ignored your robots.txt.

When you are ready to write rules, our robots.txt rule generator builds a fragment from your choices in the browser. It does not fetch your site or prove that a crawler can get through.

Sources

Sources were reviewed on 2026-10-23, except the Bing post, which did not load that day; the Bing statement was last checked against it on 2026-10-06. The OpenAI, Anthropic, Perplexity and Google rules and user-agent strings in the tables above were re-checked on 2026-11-10. Operators change their crawlers and wording, so check the linked documentation before changing a live robots.txt file.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot. OpenAI says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.

Does Google-Extended affect AI Overviews or AI Mode?

No. Google describes Google-Extended as a product token, not a separate crawler, and says it does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode use pages that Googlebot crawled and that are eligible to show a snippet.

Can robots.txt stop a user-initiated AI fetch?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because a person asked for the page. Protect private content with authentication or server rules instead.

How quickly do robots.txt changes take effect?

It depends on the operator. OpenAI says it can take about 24 hours for its systems to adjust after a robots.txt update, and Perplexity says up to 24 hours. Check your server or CDN logs after any change rather than assuming it applied immediately.