Field guides / Sources linked

AI search visibility,
without the guesswork.

Practical guides to AI crawlers, robots.txt policies and the technical checks behind being found and cited by answer engines. Each one links to the operators' own documentation.

Get cited

Make your pages accessible and useful as sources across answer engines.

Track mentions

Brand mentions & AI share of voice: sample prompts weekly and keep mention counts and citation rates.

Crawlers & access

Check crawler permissions, indexing and page signals before interpreting an answer sample.

Updated 2026-11-10

AI crawler list: user agents, purposes and robots.txt rules

Which AI crawlers fetch your site, what each one is for, and robots.txt rules that keep you visible in AI search while opting out of model training.

Updated 2026-11-10

GPTBot vs OAI-SearchBot: which OpenAI crawler to allow

GPTBot collects training data while OAI-SearchBot powers ChatGPT search. See what each OpenAI bot does, how to block one but not the other, and when changes apply.

Updated 2026-10-06

Is my website visible to AI search? A 15-minute check

Check robots.txt, CDN bot rules, noindex, JavaScript rendering and Google and Bing index coverage to see whether AI answer engines can reach and read your pages.

Updated 2026-10-06

Cloudflare AI Crawl Control: allow AI search, block training

How Cloudflare's AI crawler blocking, managed robots.txt and bot settings interact, and how to stay visible in AI answers while declining model training.

Updated 2026-10-09

GEO audit checklist: 26 checks for AI search visibility

A free GEO audit checklist and template: 26 checks for AI crawler access, indexing, structured data and content, with how each is scored and fixed.

Updated 2026-10-23

Schema markup for AI search: what helps and what does not

Do you need schema markup for AI search, AI Overviews or AI crawlers? What Google's guidance says, which JSON-LD types are worth adding and what to skip.

Updated 2026-11-10

Shopify robots.txt for AI crawlers: edit robots.txt.liquid

Allow or block GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot on Shopify by editing robots.txt.liquid, with copyable Liquid and a safe way to roll back.

Updated 2026-10-09

Block AI crawlers in WordPress and stay in AI search

Block GPTBot, ClaudeBot and other AI training crawlers in WordPress while keeping OAI-SearchBot and PerplexityBot, with a filter, Yoast or a robots.txt file.

Updated 2026-10-21

What is llms.txt? Format, example and whether it works

llms.txt is an optional Markdown map for AI tools, not a ranking signal or access control. The format, an example, who publishes one and what Google says.

Updated 2026-10-21

ChatGPT-User vs OAI-SearchBot: user agents and robots.txt

OAI-SearchBot crawls for ChatGPT search; ChatGPT-User fetches a page when someone asks. Their user agents, robots.txt rules, IP lists and how to tell them apart.

Updated 2026-10-21

What is OAI-AdsBot? OpenAI's ChatGPT ads crawler

OAI-AdsBot checks landing pages submitted as ChatGPT ads. Its user agent, IP list and robots.txt rules, and how to stop firewalls and CDNs failing your ad review.

Updated 2026-10-23

What is GPTBot? User agent, IP ranges and robots.txt

GPTBot is OpenAI's crawler for model training data. See its current user agent and IP list, robots.txt rules to block or allow it, and how to verify a visit.

Original research

Industry playbooks

Tool comparisons