A rule is an instruction, not a measurement
The generated text describes your intended access policy. It does not prove that a crawler can pass a CDN rule, a login wall or a server error. This tool does not impersonate a verified crawler, bypass a challenge, or establish indexing or citations.
Search and training serve different purposes
Keep the purpose shown beside each crawler in mind. Blocking GPTBot or ClaudeBot opts out of model training without removing you from ChatGPT or Claude search, which use OAI-SearchBot and Claude-SearchBot. Permitting a training crawler is a site-owner choice and earns no technical score points in our audit. Our AI crawler list and GPTBot vs OAI-SearchBot guide explain each token; review the provider’s documentation and your existing file before changing a live policy.
Worked example: how one file reads for each crawler
This is the kind of result the free checker returns for a live robots.txt. Below, the robots evaluator used by our audit was run on 2026-11-10 against the sample file shown here, for the path /pricing and the path /admin/login. It is a sample file, not a real site.
User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Allow: / Disallow: /admin/ User-agent: GPTBot User-agent: ClaudeBot User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: / User-agent: * Allow: / Disallow: /admin/
| Crawler | Purpose | /pricing | /admin/login | Why |
|---|---|---|---|---|
| OAI-SearchBot | ChatGPT search | Allowed | Blocked | Allow: / in the OAI-SearchBot group matches /pricing. |
| GPTBot | OpenAI training | Blocked | Blocked | Disallow: / in the GPTBot group matches /pricing. |
| Claude-SearchBot | Claude search | Allowed | Blocked | Allow: / in the Claude-SearchBot group matches /pricing. |
| ClaudeBot | Anthropic training | Blocked | Blocked | Disallow: / in the ClaudeBot group matches /pricing. |
| PerplexityBot | Perplexity search | Allowed | Blocked | Allow: / in the PerplexityBot group matches /pricing. |
| Google-Extended | Gemini training control | Blocked | Blocked | Disallow: / in the Google-Extended group matches /pricing. |
| Googlebot | Google Search | Allowed | Blocked | Allow: / in the * group matches /pricing. |
| Bingbot | Bing search | Allowed | Blocked | Allow: / in the * group matches /pricing. |
| CCBot | Common Crawl | Blocked | Blocked | Disallow: / in the CCBot group matches /pricing. |
| Bytespider | ByteDance collection | Allowed | Blocked | Allow: / in the * group matches /pricing. |
| Applebot-Extended | Apple training control | Blocked | Blocked | Disallow: / in the Applebot-Extended group matches /pricing. |
Read it this way: the search crawlers are allowed on public pages and kept out of /admin/, the training crawlers and control tokens are blocked everywhere, and Googlebot and Bingbot follow the * group because they have no group of their own. A rule result is not proof of delivery: a CDN or firewall can still stop an allowed crawler.
Guides that use these rules
- AI crawler list: user agents, purposes and robots.txt rules explains every token and which line of a robots.txt reaches it.
- Shopify robots.txt for AI crawlers shows how to add these groups with
robots.txt.liquid. - GPTBot vs OAI-SearchBot compares OpenAI’s training and search crawlers and the three policies you can choose.
Merge with your current robots.txt
Specific rules and matching groups affect the final result. Preserve your existing exclusions for private paths, check the full file and use the relevant provider’s validation tools before publishing. On Shopify or WordPress, follow the platform steps in our Shopify and WordPress guides.
Check what your live site allows
The free AI visibility checker reads your published robots.txt and reports, crawler by crawler, which search and training bots it allows for one page, alongside indexing, structured data and content checks.
Run a free AI visibility check ↗