Field guide / Updated 2026-10-06

GPTBot vs OAI-SearchBot: which OpenAI crawler to allow

Blocking GPTBot keeps your pages out of OpenAI model training. It does not take you out of ChatGPT search, which depends on a different crawler, OAI-SearchBot. Here is how OpenAI's bots divide the work and how to write rules for the policy you actually want.

The short answer

To appear in ChatGPT search answers, allow OAI-SearchBot in robots.txt and make sure your CDN or firewall lets it through.
To keep content out of OpenAI model training, disallow GPTBot.
You can do both at once. The two settings are independent, and OpenAI documents them as separate crawlers with separate IP ranges.

What each OpenAI bot does

OpenAI's crawler documentation lists four agents. The purposes below paraphrase OpenAI's own descriptions.

Token	Purpose	robots.txt	IP list
`OAI-SearchBot`	Surfaces websites in ChatGPT's search features	Applies	`https://openai.com/searchbot.json`
`GPTBot`	Crawls content that may be used to train OpenAI's generative AI foundation models	Applies	`https://openai.com/gptbot.json`
`ChatGPT-User`	Visits pages for certain user actions in ChatGPT and custom GPTs	May not apply	`https://openai.com/chatgpt-user.json`
`OAI-AdsBot`	Validates the safety of pages submitted as ads on ChatGPT	Not a crawl control	`https://openai.com/adsbot.json`

Two statements in that documentation matter most for site owners. First, OpenAI recommends allowing OAI-SearchBot in robots.txt, and allowing requests from its published IP ranges, to help your site appear in search results. Second, sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, although they can still appear as navigational links.

Three policies and their robots.txt

Pick the policy you want, then copy the matching rules. Keep any existing private-path rules: a bot that has its own group ignores the User-agent: * group, so repeat those paths inside it.

Visible in ChatGPT search, no training

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Allow both search and training

User-agent: OAI-SearchBot
User-agent: GPTBot
Allow: /

You can also leave both bots out entirely if your User-agent: * group already allows them. Naming them makes the intention explicit to anyone who reads the file later.

Opt out of both

User-agent: OAI-SearchBot
User-agent: GPTBot
Disallow: /

This removes the site from ChatGPT search answers as well as training. Choose it deliberately, not as a side effect of a broad "block AI" setting.

Check the layers below robots.txt

robots.txt is only one gate. A crawler that robots.txt allows can still fail at the network layer.

CDN and firewall rules. Cloudflare and other providers can block AI crawlers by category. Make sure the rule that blocks training bots does not also catch OAI-SearchBot. Our Cloudflare AI Crawl Control guide walks through the settings.
IP allowlists. If you only admit known crawler IPs, add the ranges from https://openai.com/searchbot.json and refresh them regularly.
Server errors and challenges. A page that returns 403, 429 or a JavaScript challenge to crawlers cannot be indexed, whatever robots.txt says.
Timing. OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust. Wait a day before concluding that a change did not work.

To confirm a visit really came from OpenAI, match the source IP against the published list. A user-agent string alone is easy to fake.

What allowing OAI-SearchBot does not do

Access is a prerequisite, not a ranking factor. Allowing the crawler means ChatGPT search can find and read your pages. Whether it cites them still depends on the question, the competing sources and how clearly your page answers it. Our technical checklist for ChatGPT citations covers the next steps, and the AI crawler list covers Anthropic, Perplexity, Google and Apple.

Be equally careful with measurement. A single answer from ChatGPT is a sample, not a trend. Keep a fixed set of prompts, record which URLs are cited, and repeat the check over several weeks before drawing conclusions.

Sources

Reviewed on 2026-10-06. OpenAI updates its user-agent versions from time to time; the tokens used in robots.txt have stayed the same.

Frequently asked questions

Should I block GPTBot?

That is a business decision about model training, not a search decision. Blocking GPTBot tells OpenAI not to use your content for training foundation models. If you also want to appear in ChatGPT search answers, keep OAI-SearchBot allowed.

What happens if I block OAI-SearchBot?

OpenAI says sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Does robots.txt control ChatGPT-User?

Not reliably. OpenAI says ChatGPT-User is initiated by a user, so robots.txt rules may not apply. It visits a page when a person's request in ChatGPT or a custom GPT needs it.

If I allow both bots, will OpenAI crawl my site twice?

Not necessarily. OpenAI says that when a site allows both GPTBot and OAI-SearchBot, it may use the results from one crawl for both purposes to avoid duplicate crawling.