Which top sites block AI crawlers?
2026 robots.txt data.

Key findings

• 19.9% of sites with a robots.txt file block at least one AI training crawler or control token at the root; 11.4% block at least one AI search crawler (OAI-SearchBot, Claude-SearchBot or PerplexityBot).
• GPTBot is blocked on 874 sites (16.0%); 685 of them name GPTBot in their own group, and the rest block it through User-agent: *.
• Of the 874 sites that block GPTBot, 454 (51.9%) still allow OAI-SearchBot, the crawler behind ChatGPT search, at the site root. For Anthropic, 411 of 821 sites that block ClaudeBot still allow Claude-SearchBot.
• 858 of the 5,949 sites that gave a definite answer for /llms.txt (14.4%) serve a file that starts with a Markdown H1, the format llmstxt.org describes.
• In the top 1,000, 21.7% of sites with a robots.txt file block GPTBot; for ranks 1,001 to 10,000 the share is 15.4%.
• 208 robots.txt files (3.8%) carry a Content-Signal line; 143 of them set ai-train=no.

These are statements in robots.txt files, not measurements of crawler behavior. A site can allow a crawler in robots.txt and still block it at its CDN or firewall, and the reverse.

How often each crawler is blocked

Share of the 5,467 sites with a parsed robots.txt file whose rules disallow the site root (/) for each token, whether through a group that names it or through User-agent: *. Tokens are grouped by what their operators say they are for; see our AI crawler list for each operator's documentation.

Google-Extended and Applebot-Extended are control tokens rather than crawlers: they govern how Google and Apple may use pages that Googlebot and Applebot collect. Read them next to the baseline rows, since a site that blocks Googlebot or Applebot is not crawled by that operator at all.

Part of this blocking comes from a CDN setting rather than rules a site wrote itself. When a site turns on Cloudflare's managed robots.txt, Cloudflare prepends a block that disallows a list of AI crawlers. 75 of the parsed files (1.4%) contain it. In all of them GPTBot, ClaudeBot, Applebot-Extended, CCBot, Meta-ExternalAgent and Bytespider are blocked; other tokens are blocked in only part of these files, as the full table's Cloudflare column shows. These files account for 75 of the 874 sites that block GPTBot.

Training versus search

Blocking a training crawler does not remove a site from the matching search product; the two are separate tokens. Counts below are sites with a parsed robots.txt file, with the share of all 5,467 such sites.

User-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) are blocked on 11.2% of these sites for at least one of the three. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it, so these rules may not stop those visits.

Top 1,000 against the rest

Crawler shares use each tier's sites with a parsed robots.txt file; the llms.txt share uses each tier's definite llms.txt answers.

llms.txt adoption

We requested /llms.txt only where robots.txt answered and allowed our crawler. A file counts as llms.txt when the request ends at /llms.txt, the body is not an HTML page and its first line is a Markdown H1. Google says its search does not use llms.txt; see does llms.txt work?

Among sites with a parsed robots.txt file and a definite llms.txt answer, 8.0% of those with an llms.txt block at least one training token, against 16.4% of those without one.

Content-Signal lines

Cloudflare's content signals add a Content-Signal line to robots.txt with yes or no for search (building a search index, not AI summaries), ai-input (feeding content to AI models for real-time answers) and ai-train (training or fine-tuning models). Cloudflare's managed robots.txt sets search=yes, ai-train=no. 208 files carry a line with at least one of the three:

Full table

All counts are sites with a parsed robots.txt file (5,467). Named: a group names the token. Blocked: the root is disallowed. Own group: blocked by a group that names the token. In a Cloudflare file: blocked on a site whose robots.txt contains Cloudflare's managed block. Partial: the root is allowed but at least one non-empty Disallow line applies.

Method

Sample

The top 10,000 registrable domains of the Tranco list 746PX, generated on 2026-10-21. The list combines five popularity rankings (Chrome UX Report, Farsight, Majestic, Cloudflare Radar and Cisco Umbrella) over the preceding 30 days with the Dowdall rule and keeps registrable domains only. Many entries are infrastructure domains, such as CDNs, DNS and APIs, that serve no website; they show up below as domains that did not answer. Our crawler never requests Google search hosts (google.com and its country domains) or the AI chat interfaces its URL guard refuses; in this list those are chatgpt.com, claude.ai and perplexity.ai.

Requests

Between 2026-10-22 23:12 and 2026-10-23 00:15 UTC our crawler requested https://<domain>/robots.txt, following up to five redirects. When the domain failed at the DNS, connection or TLS level or timed out, it tried https://www.<domain>/robots.txt once and used that answer instead. When the domain answered itself, without a redirect, with an HTML page or a 4xx status other than 401, 403 and 429, it also tried www once and used the www file only if www served one. It then requested /llms.txt from the host it read robots.txt from, only when robots.txt gave a clear answer that allows our token CiteVisorResearch: a file whose rules allow /llms.txt, or an HTML page or one of those 4xx statuses, which leave no rules. It requested nothing else. The user agent was CiteVisorResearch/1.0 (+https://citevisor.com/bot); requests to one host were at least a second apart, and network failures, timeouts, 429 and 5xx answers were retried once, at least 15 minutes after the first attempt. See our crawler page.

The first pass misread some answers: a few servers compressed their files although our client asked for no compression, and some platforms turned HTML pages into Markdown because the client also accepted Markdown. We fixed the client and requested the 55 affected domains once more between 2026-10-23 00:25 and 2026-10-23 00:26 UTC; their rows come from that request.

Parsing

Rules were evaluated as RFC 9309 describes. A token matches a group when the user-agent value, up to its first character that is not a letter, hyphen or underscore, equals the token ignoring case; there is no prefix matching. All groups naming a token are merged, and they replace the * group for that token. The longest matching path wins, counting * and $ as Google's examples do, and Allow wins a tie. Lines that do not parse are skipped. Files were read up to 512 KiB; 0 were longer.

What was fetched

Of the 2,085 domains with no answer, 1,389 failed at DNS, 314 at TLS, 279 timed out, 24 refused or dropped the connection, 15 redirected more than five times, 61 pointed to an address or redirect target our URL guard refuses, such as a non-public address or a Google search or AI chat host, and 3 failed in other ways. 2,454 domains failed at the network level and were tried again at www.<domain>; 27 served no robots.txt file themselves but did at www.<domain>. 3,039 parsed files were reached through a redirect; 5,260 parsed files contain at least one user-agent group, and 234 disallow the root in their User-agent: * group, which binds every crawler without a group of its own.

Limitations

• One list, one snapshot. The top 10,000 of one ranking is not the whole web, and robots.txt files change often.
• One network location. Some sites refuse unfamiliar clients or answer differently by region; refusals, rate limits, server errors and failed connections are counted separately and left out of every share.
• Statements, not behavior. robots.txt does not show what crawlers actually do, or what a CDN or firewall blocks.
• Our parser follows RFC 9309. Older parsers that match by prefix, or that accept misspelled directives, may read a few files differently.
• The llms.txt check looks at the root path, the content type and the first heading only. The proposal also allows a file under a subpath such as /docs/llms.txt, which this study does not count.
• Categories follow each operator's documentation. ByteDance publishes no Bytespider documentation we could find, and Anthropic's current crawler article no longer lists anthropic-ai, so both sit under other AI tokens.
• Google search hosts and the AI chat interfaces our URL guard refuses are never fetched, so google.com, its country domains, chatgpt.com, claude.ai and perplexity.ai are counted as not requested.

Download the data

Download the CSV (one row per domain). Columns: rank, domain, robots_result (ok, html for a web page or binary file, missing, denied, rate_limited, server_error, failed or excluded, as in the table above), robots_status, robots_url (the final URL after redirects, without its query string), robots_bytes, llms_result, llms_status, content_signal, named_tokens, then one column per token with blocked, partial or allowed; a token column is empty when robots.txt gave no usable answer. The data is free to reuse under CC BY 4.0 with a link to this page. To have a domain removed, email contact@citevisor.com; removals are listed here. No rows have been removed so far.

Sources

• RFC 9309: Robots Exclusion Protocol
• Google: How Google interprets the robots.txt specification
• Le Pochat et al., Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation (NDSS 2019), and list 746PX
• The /llms.txt file proposal
• Cloudflare: managed robots.txt and content signals
• Meta: web crawlers, including Meta-ExternalAgent
• Anthropic: how site owners can block its crawlers
• AI crawler list: user agents, purposes and robots.txt rules, with each operator's documentation