Shopify robots.txt for AI crawlers: edit robots.txt.liquid

What Shopify's default robots.txt does with AI crawlers

Open https://your-store.com/robots.txt. On the default Shopify files we inspected on 2026-10-09, the structure was:

• a User-agent: * group that allows the storefront (Allow: /) and disallows private and transactional paths such as /admin, /cart/, /checkout, /checkouts/, /orders and /account, plus some filter and preview parameters;
• a separate User-agent: adsbot-google group repeating the checkout and order rules;
• one or more Sitemap: lines.

None of the AI crawlers had a group of their own. Under the robots exclusion standard, a crawler without a named group follows the * group. So by default OAI-SearchBot, Claude-SearchBot, PerplexityBot, GPTBot and ClaudeBot may all crawl your products, collections, pages and blog, and all stay out of checkout.

That means you only need to edit anything if you want to block some of them. If your goal is to appear in AI search, the default already lets the search crawlers in; check your CDN, apps and theme for anything that overrides it.

Before you edit

Shopify's help center is clear about the risk: editing robots.txt.liquid is an unsupported customization, Shopify Support cannot help with it, and incorrect use can result in loss of all traffic. Shopify also notes that a theme with an older robots.txt.liquid template can keep rules the current default no longer has, such as Disallow: /search and Disallow: /policies/.

Decide your policy first. Training crawlers and search crawlers are separate tokens, so you can opt out of model training without leaving AI search. Our AI crawler list explains each one.

Step by step: create robots.txt.liquid

Shopify's documented steps, as of the review date:

1. From your Shopify admin, go to Online Store.
2. For the relevant theme, open the theme's menu and choose Edit code.
3. Click Add a new template, then select robots.
4. Click Create template.
5. Edit the template, then save it in your published theme.

The template: keep Shopify's rules, add AI crawler groups

Shopify recommends keeping its Liquid objects so you inherit future default changes. Rules that are not part of a default group go outside the Liquid loop. This template keeps every default group and adds groups that block training crawlers while leaving search crawlers on the * rules:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Do not add a group such as User-agent: OAI-SearchBot with Allow: / "for good measure". A crawler with its own group ignores the * group entirely, so that group would drop Shopify's checkout, cart and account exclusions for that bot. Leave search crawlers on the default rules.

If you decide to leave AI search altogether, add Disallow: / groups for OAI-SearchBot, Claude-SearchBot and PerplexityBot as well. Expect your store to stop appearing in those engines' answers.

Check the result

1. Open https://your-store.com/robots.txt in a private window and confirm the default groups are still there, followed by your new groups.
2. Use the robots.txt report in Google Search Console to confirm Google fetched the new file without errors.
3. Run the free AI visibility checker on a product page. It reports, crawler by crawler, which of 11 AI and search bots your published robots.txt allows for that page.

You can also build the custom groups with our browser-only robots.txt generator for AI crawlers and paste them below the Liquid loop.

Roll back

From your Shopify admin, go to Online Store, open Edit code, right-click robots.txt.liquid, select Delete permanently and confirm. Shopify serves the default robots.txt again.

Limits of robots.txt

robots.txt is a request that documented crawlers follow, not an access control. It does not stop a bot that ignores it, and agents acting for a person, such as ChatGPT-User, may not apply robots.txt rules at all according to OpenAI. Changes also take time: crawlers re-fetch robots.txt on their own schedule.

Sources

• Shopify Help Center: Editing robots.txt.liquid
• Shopify developer docs: Customize robots.txt
• Shopify developer docs: robots.txt.liquid template
• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl the web, and how can site owners block the crawler?
• Google: Google's common crawlers, including Google-Extended
• RFC 9309: Robots Exclusion Protocol

Reviewed on 2026-10-09. The default robots.txt description comes from public Shopify storefront files inspected that day; Shopify can change its defaults, so check your own file.