Training Crawlers vs Search Crawlers: The robots.txt Rules That Actually Affect Visibility

Most teams treating robots.txt as "block AI or allow AI" are making a binary decision on a problem that has at least six settings.

AI companies now run separate crawlers for training and for search retrieval. Blocking GPTBot stops OpenAI from adding your content to training data — but it does not necessarily block OAI-SearchBot, which feeds ChatGPT's live search index. Block the wrong bot and you lose AI visibility while thinking you only opted out of training.

This article maps the crawler taxonomy and gives you the robots.txt rules that actually affect whether AI systems cite your brand.

The crawler taxonomy

Crawler	Company	User-agent	What it does
GPTBot	OpenAI	`GPTBot`	Crawls for training corpus
OAI-SearchBot	OpenAI	`OAI-SearchBot`	Indexes for ChatGPT search / retrieval
ChatGPT-User	OpenAI	`ChatGPT-User`	Live fetches triggered by user queries
ClaudeBot	Anthropic	`ClaudeBot`	Crawls for Claude training and retrieval
Google-Extended	Google	`Google-Extended`	Controls Gemini/Vertex AI training use
Googlebot	Google	`Googlebot`	Standard search + AI Overviews indexing
PerplexityBot	Perplexity	`PerplexityBot`	Indexes for Perplexity citations
Applebot-Extended	Apple	`Applebot-Extended`	Apple Intelligence training
CCBot	Common Crawl	`CCBot`	Feeds downstream AI datasets

The critical distinction: training crawlers build model knowledge. Search crawlers build retrieval indexes that determine what gets cited in live answers. They are governed by separate user-agent tokens in robots.txt.

The four configurations that matter

1. Maximum AI visibility (recommended for most brands):

Allow all search crawlers. Optionally block training-only crawlers if you want to opt out of model training without losing citations.

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

This lets ChatGPT, Perplexity, and Google AI search/retrieve your content while opting out of training use.

2. Full AI opt-out (visibility and training):

Block everything. Your content will not appear in AI answers. Use only if you have a deliberate reason — competitive sensitivity, licensing restrictions, or regulatory requirements.

3. Google-only visibility:

Allow Googlebot (required for search and AI Overviews). Block Google-Extended only if you want to opt out of Gemini training while keeping AI Overviews. Note: blocking Googlebot removes you from both organic search and AI Overviews.

4. Accidental block (the silent killer):

Many sites inherit blocks they did not intend:

Cloudflare's "block AI bot traffic" setting adds AI crawler blocks by default
WordPress security plugins that blanket-block unknown bots
Copied robots.txt templates from privacy-focused sites
Staging environment rules that leaked to production

Ahrefs' Site Audit now flags AI crawler blocks specifically. Check yours.

What robots.txt does not control

Already-scraped content. Blocking a crawler today does not remove content already in a training corpus or index.
Non-compliant scrapers. Rogue bots ignore robots.txt entirely. Real enforcement requires WAF/CDN rules.
llms.txt. This is a content map, not an access-control file. Major consumer AI search products do not officially consume it. Use it for developer documentation if helpful; do not rely on it for visibility.
Third-party mentions. AI systems cite Reddit, G2, and press coverage about your brand regardless of your robots.txt.

Common mistakes

Blocking all AI bots to "protect content." You protect content from training while simultaneously making yourself invisible in the AI answers your buyers use to discover products. The training/search split exists precisely so you do not have to choose.

Blocking Google-Extended thinking it blocks AI Overviews. It does not. Googlebot controls AI Overviews indexing. Google-Extended only controls Gemini/Vertex training use.

Never checking after a platform migration. CMS changes, CDN switches, and plugin updates frequently reset crawler rules.

How this connects to your AEO strategy

If Obsurfable or any monitoring tool shows zero citations despite strong content, robots.txt is the first technical check. A single Disallow line for OAI-SearchBot explains an entire visibility gap.

For the broader technical picture, see Is Your robots.txt Hurting Your AI Visibility? and What Is llms.txt?.

The bottom line

AI crawler control is not one switch. Training bots and search bots are separate, and blocking the wrong one silently kills your visibility. Audit your robots.txt for the specific user-agents that feed answer-engine retrieval — not just the ones that feed model training.