CiteVisorAI search readinessAudit / Coming soon
Home / Crawler rules
Free tool / Works in your browser

Clear rules.
Deliberate access.

Generate a robots.txt fragment for 11 crawlers from your own permission choices. This tool runs locally in your browser; it does not fetch a website or submit your choices to a server.

Browser-only rule generatorNo website fetching

Choose each permission.

Checked means allow; unchecked means disallow. Search crawlers start allowed, while training and collection controls start disallowed. Review each purpose for your site.

Suggested robots.txt fragment
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

You can also select and copy the text. Existing groups and more-specific rules may change the result; do not blindly replace your file.

A rule is an instruction, not a measurement

The generated text describes your intended access policy. It does not prove that a crawler can pass a CDN rule, a login wall or a server error. This tool does not impersonate a verified crawler, bypass a challenge, or establish indexing or citations.

Search and training serve different purposes

Keep the purpose shown beside each crawler in mind. Permitting a training crawler is a site-owner choice and earns no technical score points in our audit. Review the provider’s documentation and your existing file before changing a live policy.

Merge with your current robots.txt

Specific rules and matching groups affect the final result. Preserve your existing exclusions for private paths, check the full file and use the relevant provider’s validation tools before publishing.

Join the audit waitlist ↗