What is GPTBot? User agent, IP ranges and robots.txt

The short answer

• What it is: OpenAI's web crawler for content that may be used to train its generative AI foundation models. Its robots.txt token is GPTBot.
• How to block it: add a User-agent: GPTBot group with Disallow: /. OpenAI says disallowing GPTBot indicates that a site's content should not be used in training.
• What blocking does not change: whether you can appear in ChatGPT search, which depends on OAI-SearchBot, or how you rank in Google, whose crawlers do not read GPTBot's rules.
• How to verify a visit: match the source IP against https://openai.com/gptbot.json. A user-agent string on its own is easy to copy.

GPTBot's user agent

OpenAI's crawler documentation gives this example user-agent string and notes that the version number may change:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

That was the current version when we checked on 2026-10-23. Many older articles quote GPTBot/1.0 or GPTBot/1.1. The version changes; the robots.txt token does not. Match on GPTBot in log filters and firewall rules rather than on the full string, so that a version bump does not break them.

OpenAI also says it may add a robots.txt marker to the user-agent string when it fetches robots.txt files, to help site owners tell those requests apart when their logs do not record paths. Treat such a request like any other GPTBot visit: check the IP before trusting it.

What GPTBot does and does not do

OpenAI runs several agents, each with its own job. GPTBot is only one of them.

Three details from OpenAI's documentation matter when you decide what to do about GPTBot:

• The settings are independent. You can disallow GPTBot and allow OAI-SearchBot. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.
• One crawl can serve both purposes. If your site allows both GPTBot and OAI-SearchBot, OpenAI may use the results of a single crawl for both, to avoid duplicate crawling. A quiet GPTBot line in your logs does not mean your pages are out of training.
• The rule speaks about training use. OpenAI's documentation says what a GPTBot disallow indicates for training. It says nothing about content collected before you added the rule.

The other OpenAI agents have their own guides: GPTBot vs OAI-SearchBot, ChatGPT-User vs OAI-SearchBot and OAI-AdsBot.

Block or allow GPTBot in robots.txt

Pick the policy you want and copy the group. Under the robots.txt standard, RFC 9309, a crawler that finds a group naming it obeys that group and ignores the User-agent: * group. Repeat any private-path rules inside the GPTBot group.

Block GPTBot from the whole site

User-agent: GPTBot
Disallow: /

Keep some sections out of training

User-agent: GPTBot
Disallow: /members/
Disallow: /research/

Paths that no rule in the group matches stay open to GPTBot.

Allow only one section

User-agent: GPTBot
Allow: /docs/
Disallow: /

This works because the most specific matching rule wins: /docs/ is longer than /, so pages under /docs/ are allowed and the rest of the site is not. If an allow rule and a disallow rule are equally specific, the standard says the allow rule should be used.

Stay in ChatGPT search, opt out of training

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Two matching details catch people out. The user-agent token is matched without regard to case, so gptbot and GPTBot name the same group. Paths are matched with case, so Disallow: /Members/ does not cover /members/.

For platform steps, see our guides for WordPress, Shopify and Cloudflare. The rule generator writes a fragment for 11 AI and search crawlers in your browser.

Check the layers in front of robots.txt

robots.txt is a set of instructions for crawlers that choose to follow them. In its guidance for advertisers, OpenAI says its crawlers respect robots.txt and stop crawling immediately when access is disallowed. Your server and CDN decide what any request actually receives.

1. If you want enforcement, a CDN or firewall rule can refuse GPTBot at the edge. Write the rule for GPTBot alone. A broad "AI bots" category can also catch OAI-SearchBot and take you out of ChatGPT search.
2. If you want GPTBot allowed, make sure bot protection does not challenge it. A crawler that receives a 403, a 429 or a JavaScript challenge cannot read the page, whatever robots.txt says.
3. Give changes a day. RFC 9309 says crawlers should not use a cached robots.txt for more than 24 hours unless the file is unreachable. OpenAI says search results can take about 24 hours to reflect a robots.txt update and gives no separate timing for GPTBot.

Verify that a visit is really GPTBot

Anyone can send a request with GPTBot's user-agent string, so confirm the source IP before acting on log data. In a standard access log the client IP is the first field:

grep -i "gptbot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head

Then check each address against OpenAI's published list:

python3 - 203.0.113.7 <<'EOF'
import ipaddress, json, sys, urllib.request
data = json.load(urllib.request.urlopen("https://openai.com/gptbot.json"))
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix")) for p in data["prefixes"]]
ip = ipaddress.ip_address(sys.argv[1])
print("in gptbot.json" if any(ip in net for net in nets) else "not in gptbot.json")
EOF

When we checked on 2026-10-23, the file listed 18 IPv4 prefixes and carried a creation time of 2026-09-22. OpenAI warns that its crawler infrastructure may change over time, so fetch the list when you check rather than relying on addresses you once saw in your logs.

A request that matches the list and keeps fetching paths you disallowed more than a day earlier is worth keeping as evidence: note the timestamp, path, status code and IP.

Should you block GPTBot?

It is a business decision, and it is separate from search visibility. Common reasons to block it are content you license or sell, member-only or paywalled material, and a policy against model training. Common reasons to allow it are public documentation, help articles and product facts that you would like AI models to describe accurately. Nobody can promise that allowing GPTBot leads to mentions in ChatGPT answers, and OpenAI's documentation makes no such claim.

Whatever you choose, make the decision explicit. Name GPTBot in robots.txt instead of relying on a CDN's broad AI category, and keep OAI-SearchBot allowed if you want to appear in ChatGPT search. Training crawlers from other companies, such as ClaudeBot, CCBot and Google's Google-Extended token, are covered in the AI crawler list.

Sources

• OpenAI: Overview of OpenAI crawlers
• OpenAI IP list for GPTBot
• OpenAI Help: Advertiser guidance for allowing OpenAI web crawlers
• RFC 9309: Robots Exclusion Protocol
• Google: Google's common crawlers and Google-Extended

Reviewed on 2026-10-23. OpenAI changes the version number in GPTBot's user agent from time to time; the robots.txt token stays the same.