AI search visibility,
without the guesswork.
Practical guides to AI crawlers, robots.txt policies and the technical checks behind being found and cited by answer engines. Each one links to the operators' own documentation.
AI crawler list: user agents, purposes and robots.txt rules
Which AI crawlers fetch your site, what each one is for, and robots.txt rules that keep you visible in AI search while opting out of model training.
Updated 2026-10-06GPTBot vs OAI-SearchBot: which OpenAI crawler to allow
GPTBot collects training data while OAI-SearchBot powers ChatGPT search. See what each OpenAI bot does, how to block one but not the other, and when changes apply.
Updated 2026-10-06How to get cited by ChatGPT search: a technical checklist
No one can guarantee a ChatGPT citation. This checklist covers what you control: crawler access, indexable answer-first pages, verifiable facts and measurement.
Updated 2026-10-06Is my website visible to AI search? A 15-minute check
Check robots.txt, CDN bot rules, noindex, JavaScript rendering and Google and Bing index coverage to see whether AI answer engines can reach and read your pages.
Updated 2026-10-06Cloudflare AI Crawl Control: allow AI search, block training
How Cloudflare's AI crawler blocking, managed robots.txt and bot settings interact, and how to stay visible in AI answers while declining model training.
Updated 2026-10-02Does llms.txt work? Separate navigation from rankings
What the llms.txt proposal is for, what Google's guidance says, and why it is an optional experiment rather than a route into AI answers.