Free tool / Works in your browser

Robots.txt rules
for AI crawlers.

Generate a robots.txt fragment for 11 crawlers from your own permission choices. This tool runs locally in your browser; it does not fetch a website or submit your choices to a server.

Checking a live site instead? The free AI visibility checker fetches your published robots.txt and shows the verdict for each of these 11 crawlers on the page you enter, with the matching rule, and no email needed. It reads robots.txt rules only: it cannot see firewall or CDN rules that apply to the real crawlers’ IP addresses.
Browser-only rule generatorNo website fetching

Choose each permission.

Checked means allow; unchecked means disallow. Search crawlers start allowed, while training and collection controls start disallowed. Review each purpose for your site.

Suggested robots.txt fragment
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

You can also select and copy the text. Existing groups and more-specific rules may change the result; do not blindly replace your file.

A rule is an instruction, not a measurement

The generated text describes your intended access policy. It does not prove that a crawler can pass a CDN rule, a login wall or a server error. This tool does not impersonate a verified crawler, bypass a challenge, or establish indexing or citations.

Search and training serve different purposes

Keep the purpose shown beside each crawler in mind. Blocking GPTBot or ClaudeBot opts out of model training without removing you from ChatGPT or Claude search, which use OAI-SearchBot and Claude-SearchBot. Permitting a training crawler is a site-owner choice and earns no technical score points in our audit. Our AI crawler list and GPTBot vs OAI-SearchBot guide explain each token; review the provider’s documentation and your existing file before changing a live policy.

Worked example: how one file reads for each crawler

This is the kind of result the free checker returns for a live robots.txt. Below, the robots evaluator used by our audit was run on 2026-11-10 against the sample file shown here, for the path /pricing and the path /admin/login. It is a sample file, not a real site.

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /admin/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/
CrawlerPurpose/pricing/admin/loginWhy
OAI-SearchBotChatGPT searchAllowedBlockedAllow: / in the OAI-SearchBot group matches /pricing.
GPTBotOpenAI trainingBlockedBlockedDisallow: / in the GPTBot group matches /pricing.
Claude-SearchBotClaude searchAllowedBlockedAllow: / in the Claude-SearchBot group matches /pricing.
ClaudeBotAnthropic trainingBlockedBlockedDisallow: / in the ClaudeBot group matches /pricing.
PerplexityBotPerplexity searchAllowedBlockedAllow: / in the PerplexityBot group matches /pricing.
Google-ExtendedGemini training controlBlockedBlockedDisallow: / in the Google-Extended group matches /pricing.
GooglebotGoogle SearchAllowedBlockedAllow: / in the * group matches /pricing.
BingbotBing searchAllowedBlockedAllow: / in the * group matches /pricing.
CCBotCommon CrawlBlockedBlockedDisallow: / in the CCBot group matches /pricing.
BytespiderByteDance collectionAllowedBlockedAllow: / in the * group matches /pricing.
Applebot-ExtendedApple training controlBlockedBlockedDisallow: / in the Applebot-Extended group matches /pricing.

Read it this way: the search crawlers are allowed on public pages and kept out of /admin/, the training crawlers and control tokens are blocked everywhere, and Googlebot and Bingbot follow the * group because they have no group of their own. A rule result is not proof of delivery: a CDN or firewall can still stop an allowed crawler.

Guides that use these rules

Merge with your current robots.txt

Specific rules and matching groups affect the final result. Preserve your existing exclusions for private paths, check the full file and use the relevant provider’s validation tools before publishing. On Shopify or WordPress, follow the platform steps in our Shopify and WordPress guides.

Check what your live site allows

The free AI visibility checker reads your published robots.txt and reports, crawler by crawler, which search and training bots it allows for one page, alongside indexing, structured data and content checks.

Run a free AI visibility check ↗