Block AI crawlers in WordPress and stay in AI search

How WordPress serves robots.txt

Unless a physical file exists, WordPress answers /robots.txt itself. The core function do_robots() builds this output:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/wp-sitemap.xml

The Sitemap: line comes from WordPress's built-in sitemaps (since version 5.5) and is only added when the site is public. SEO plugins often replace it with their own sitemap URL. Everything passes through the robots_txt filter before it is sent, which is how plugins add rules.

On typical Apache and nginx setups, a real robots.txt file in the site root is served directly and the virtual output, including any filter or plugin rules, is not used.

Choose the policy first

Training and search use different user agents, so you can treat them separately:

The rest of this guide blocks the training group and leaves search alone. Our AI crawler list has each operator's own description.

Option 1: Add rules with the robots_txt filter

Put this in a small custom plugin or a child theme's functions.php. It appends one group per training crawler and leaves WordPress's own rules untouched:

add_filter( 'robots_txt', function ( $output, $public ) {
	if ( ! $public ) {
		return $output;
	}
	$training_bots = array( 'GPTBot', 'ClaudeBot', 'Google-Extended', 'CCBot', 'Applebot-Extended', 'Bytespider' );
	foreach ( $training_bots as $bot ) {
		$output .= "\nUser-agent: {$bot}\nDisallow: /\n";
	}
	return $output;
}, 20, 2 );

The $public check skips the rules when the site is set to discourage search engines. Search crawlers keep following the User-agent: * group. Do not add an Allow: / group for a search bot: a bot with its own group ignores the * group, including the /wp-admin/ rule.

Option 2: Use your SEO plugin's editor

If you already use Yoast SEO, go to Yoast SEO, then Tools, then File editor, and create or edit robots.txt there. Yoast notes the menu is hidden when file editing is disabled on the WordPress install. Other SEO plugins offer similar editors; check their documentation, because some write a physical file and some filter the virtual one. Paste the same training-bot groups below the existing rules.

Option 3: Upload a physical robots.txt

You can place a plain robots.txt in the web root by SFTP or your host's file manager. It replaces the virtual file completely, so copy WordPress's default lines and your sitemap URL into it, then add the training-bot groups. From then on, plugin and filter changes to robots.txt have no effect until you remove the file.

Don't use "Discourage search engines" for this

Settings → Reading → Search engine visibility is for staging sites. Since WordPress 5.3 it no longer outputs Disallow: /; it adds a noindex robots meta tag to your pages instead. That asks every search engine to drop the site, which also removes it from AI Overviews and other search-backed answers. It is not a training opt-out.

Hosting and CDN rules

robots.txt is a request. If you need enforcement, or a host or CDN already filters bots, check that layer too: a firewall rule that blocks OAI-SearchBot will hide you from ChatGPT search even when robots.txt allows it. If your site runs behind Cloudflare, see our guide to Cloudflare AI Crawl Control.

Verify the change

1. Clear page and CDN caches, then open https://your-site.com/robots.txt in a private window.
2. Confirm the User-agent: * group and sitemap line are still present, followed by your new groups.
3. Check the robots.txt report in Google Search Console for fetch errors.
4. Run the free AI visibility checker on a key page to see, crawler by crawler, which of 11 AI and search bots your live robots.txt allows.

Sources

• WordPress developer reference: do_robots()
• WordPress developer reference: robots_txt filter
• WordPress developer reference: WP_Sitemaps::add_robots()
• WordPress developer reference: wp_robots_no_robots()
• Yoast: The robots.txt file in Yoast SEO
• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl the web, and how can site owners block the crawler?

Reviewed on 2026-10-09. Plugin menus change between versions; check your plugin's documentation if a label differs.