Field guide / Updated 2026-10-06

Is my website visible to AI search? A 15-minute check

Before you measure how often AI assistants mention your brand, confirm the unglamorous part: their crawlers can reach your pages and read the text. Many visibility problems turn out to be ordinary access and indexing errors that take minutes to find.

Step 1: Read your robots.txt

Open https://your-site.com/robots.txt in a browser, or run curl -s https://your-site.com/robots.txt in a terminal.

Look for Disallow: / under User-agent: *. That line asks every crawler, including search engines, to stay away from the whole site.
Look for named groups such as User-agent: OAI-SearchBot or User-agent: PerplexityBot. A bot that has its own group follows only that group and ignores the * rules.
Decide which bots you want. Search crawlers such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot affect visibility. Training crawlers such as GPTBot and ClaudeBot do not affect search results in those products.

Our AI crawler list explains what each token does.

Step 2: Check your CDN and firewall

robots.txt is a request, not a gate. The real gate is often your CDN, firewall or hosting provider.

If you use Cloudflare, review AI Crawl Control and the bot settings. Since July 2025, Cloudflare has asked new domains during setup whether to allow AI crawlers. Our Cloudflare guide explains how blocking and the managed robots.txt interact.
Ask your hosting provider whether it blocks or rate-limits AI crawlers, and on which plans.
In your logs, look for 403, 429 or challenge responses served to search crawlers. Verify the source IP against the operator's published list before drawing conclusions, because user agents can be faked.

Step 3: Look for noindex and wrong canonicals

A crawler can fetch a page and still be told not to use it.

curl -sI https://your-site.com/page | grep -i -E "x-robots-tag|location|HTTP/"
curl -s https://your-site.com/page | grep -i -E "name=\"robots\"|rel=\"canonical\""

An X-Robots-Tag: noindex header or a <meta name="robots" content="noindex"> tag keeps the page out of search results, and with them out of AI features that depend on those results.
A canonical tag that points to a different URL, a staging host or the homepage tells search engines to prefer that other page.
Redirect chains and soft 404 pages waste crawl and can stop indexing.

Step 4: View the page without JavaScript

Run curl -s https://your-site.com/page | grep -c "a sentence from your page". If the count is zero, your main content probably appears only after JavaScript runs.

Google renders JavaScript, but not every crawler does so the way a browser does. Text that exists in the initial HTML is the safest choice for anything you want quoted: product descriptions, prices, FAQs, documentation. Server-side rendering or static generation solves this for most frameworks.

Step 5: Confirm index coverage in Google and Bing

Answer engines lean on search indexes, so check both major ones.

In Google Search Console, use URL Inspection on your key pages and test the live URL. Google says pages must be indexed and eligible to show a snippet to appear as a supporting link in AI Overviews and AI Mode. Clicks from those features are counted in the Performance report under Web search.
In Bing Webmaster Tools, submit your sitemap, inspect key URLs and open the AI Performance report. It shows when your pages are cited in Microsoft Copilot and Bing's AI answers.
Search engines need time. Index coverage for a new site can take days to weeks.

Step 6: Ask the engines, consistently

Once access and indexing are clean, measure what the engines actually say.

Choose 10 to 20 prompts a real customer would ask, and keep the wording fixed.
Ask them in ChatGPT, Perplexity, Gemini, Claude and Copilot, and record each answer and the cited URLs.
Repeat on a schedule. One answer is a sample; a pattern across weeks is evidence.

If an engine never mentions you while competitors are cited, compare the cited pages with yours. They often answer the question more directly, show clearer facts, or have more independent mentions. Our ChatGPT citation checklist covers those content steps.

Sources

Reviewed on 2026-10-06. Passing every step means the technical path is open; it does not guarantee that any engine will mention or cite a page.

Frequently asked questions

Why does my site not appear in ChatGPT or Perplexity answers?

Start with access. Check that OAI-SearchBot and PerplexityBot are not blocked in robots.txt or at your CDN, that the page is indexable, and that its main text is in the initial HTML. If all of that passes, the cause is more likely relevance or competition than a technical block.

Does Google Search Console show AI Overviews traffic?

Yes, as part of overall search traffic. Google says sites appearing in AI features are included in the Performance report under the Web search type. It is not broken out as a separate AI-only report.

Is there a report for Microsoft Copilot citations?

Bing Webmaster Tools has an AI Performance report, in public preview since February 2026, that shows when your pages are cited in Microsoft Copilot and Bing's AI-generated answers.

Do I need an llms.txt file to be visible?

No. Google says you do not need new machine-readable files or AI text files to appear in its AI features. llms.txt is an optional navigation aid, not an access requirement.