Google, Microsoft, and OpenAI Plan a Shared robots.txt AI Training Opt-Out — What Publishers Need to Know

On October 31, 2026, Google, Microsoft, and OpenAI each published posts saying they will honor a shared robots.txt signal for AI training preferences. Anthropic’s Kevin Kelley submitted the underlying design to the IETF AI Preferences working group the same day. Search Engine Journal’s Matt G. Southern summarized the announcements on November 1.

The pitch is simple: one rule group instead of a stack of per-vendor crawler blocks. The reality is messier — none of the three vendors supports the line yet, Google ties a shared opt-out to Google-Extended (training and Gemini Apps grounding), and Microsoft is splitting training from grounding while retiring NoArchive and NoCache for AI use cases.

If your AEO program depends on appearing in AI answers, this is not a “set and forget privacy” update. It is a map of which levers still control visibility versus model learning.

What the draft actually adds

Kelley’s draft does not invent a new robots.txt directive. It introduces a policy token user-agent, __usage_ai-training, that sites use in an ordinary rule group with Allow and Disallow.

Example from the draft (opt out of training site-wide except /press-kit/):

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

AI Training in the working-group vocabulary means using content to update a generative model’s learned parameters. Non-generative systems (classification, ranking) are out of scope. The token does not control crawling — only declared usage preferences once a participating company reads the file.

Important defaults from the draft:

A wildcard User-Agent: * group never matches the token.
A file with no __usage_ai-training group means no stated preference — which Kelley notes is true of essentially every robots.txt file published today.
Existing per-bot rules are not folded into the shared preference automatically.

The group had been working on a separate Content-Usage draft (co-authored by Google’s Gary Illyes). Kelley wrote that effort was set aside to prioritize vocabulary first — covered earlier when Search Engine Journal reported on the Content-Usage draft in October.

Vendor timelines and interpretations

Provider	Stated support	How they read a shared training opt-out
Google	Early 2027 (Mueller post; crawler docs cite early 2027)	If you opt out via the shared line and do not specify `Google-Extended`, Google plans to treat it as if you disallowed Google-Extended
Microsoft	Honor new prefs from Feb 1, 2027; stop recognizing NoArchive/NoCache for AI after Mar 1, 2027	Shared line applies to training; grounding is a separate page-level preference
OpenAI	“Next few months”	Current GPTBot policy remains until then; docs (checked Oct 30) did not resolve conflicts with existing GPTBot rules
Anthropic	Implementation “will take some time”	Kelley said development started; Anthropic’s public crawler help still lists ClaudeBot, Claude-User, Claude-SearchBot only

Until each vendor ships support, your existing bot-specific rules remain the enforceable controls — see Training Crawlers vs Search Crawlers and Is Your robots.txt Hurting AI Visibility?.

Google: shared opt-out maps to Google-Extended

John Mueller’s Google post is explicit: a shared training opt-out without a separate Google-Extended directive will be read as a Google-Extended block.

Google-Extended governs:

Use of content to train future Gemini models
Use for grounding in Gemini Apps and on Vertex AI

Grounding, in Google’s definition, is supplying content from the search index when the model answers — not the same as classic ranking, but directly tied to what Gemini can cite or summarize in app experiences.

Google also repeats that Google-Extended does not affect normal Google Search inclusion. AI Overviews and AI Mode use a separate opt-out via Search Console’s generative AI control — see Google AI Overviews auto-expand coverage for how answer surfaces diverge from classic snippets.

Strategic tension: the draft’s shared line is training-only in vocabulary, but Google’s mapping can remove grounding eligibility in Gemini Apps when you only wanted to block weight updates. There is no documented split in Google’s crawler docs to allow grounding while blocking training — unlike Microsoft’s model.

Sites that allow Google-Extended remain eligible for Gemini Apps responses through grounding, per Google’s documentation.

Microsoft: training and grounding are two switches

Microsoft’s announcement (Krishna Madhavan, Bing) adopts the shared line for training and adds a distinct AI Grounding preference at the page level. Training can also be set per page via meta tag or HTTP header, overriding robots.txt for that URL.

Grounding defaults to Allow. Disallow keeps content out of AI-generated answers. DisallowExtended is Microsoft-specific: it allows grounding in Microsoft-created tools (Copilot, Bing AI Mode) but blocks grounding through Microsoft-provided grounding services used by other businesses (Web IQ).

Microsoft states either grounding restriction limits brand visibility in AI answers but does not affect classic Bing search listings.

Migration pressure is real:

NoArchive and NoCache stop working as training/grounding controls after March 1, 2027
New preferences honored from February 1, 2027 as pages are recrawled
Sites still on old tags can set a temporary preference in Bing Webmaster Tools (opens December 15, 2026) until migration completes
Pages without a new preference by March 1 default to allowing grounding until recrawl — Microsoft’s FAQ warns explicitly

Once Microsoft detects either new preference on a site, it ignores remaining NoArchive/NoCache on that site.

For Copilot-heavy categories, this is the first major platform to formalize the distinction marketers have been guessing at: training crawlers versus answer grounding. Blocking the wrong one still silently removes you from citations — the same failure mode as blocking OAI-SearchBot while trying to block GPTBot.

OpenAI and Anthropic gaps

OpenAI pledged to honor the signal in the coming months but gave no conflict rule when __usage_ai-training and GPTBot disagree. Until documentation catches up, treat GPTBot and OAI-SearchBot separately — ChatGPT Search retrieval is not the same product surface as training ingestion.

Anthropic had no newsroom post as of October 30; Kelley’s mailing-list note is the primary source for implementation intent. Claude’s search bot (Claude-SearchBot) remains the visibility-relevant allowlist for answer-engine presence, distinct from training-oriented bots.

What this means for AI visibility strategy

1. Audit before you consolidate.

Replacing five bot blocks with one shared Disallow: / training group may be correct for privacy — and catastrophic for Gemini Apps grounding under Google’s mapping. Document intent: “no training” vs “no appearance in answers” vs “no third-party API grounding.”

2. Microsoft customers on NoArchive/NoCache need a Q4 2026 calendar item.

December 15, 2026 WMT temporary settings and February–March 2027 cutovers are fixed dates. Legal and SEO teams often own those tags; AEO owners need a seat at the table.

3. Shared line ≠ search bot access.

Nothing in Kelley’s draft replaces OAI-SearchBot, ChatGPT-User, PerplexityBot, or Googlebot. Why each AI engine reads a different search index still applies — training opt-out does not synchronize citation behavior across ChatGPT, Gemini, and Copilot.

4. Measure answers, not just crawler hits.

Policy changes alter which pages can ground answers, not just whether a model memorized your homepage. Track brand mentions and citations on buyer prompts in Obsurfable’s public corpus or your own prompt panel — crawler allowlists alone do not prove visibility.

5. Do not confuse Google’s “GEO is SEO” message with training policy.

Google’s AI optimization guide argues AEO/GEO for Search is still SEO — indexed, crawlable, useful pages. That is orthogonal to Google-Extended, which is about Gemini training and app grounding. We unpacked the guide’s limits in Google Says AEO and GEO Are Still SEO.

Practical checklist

Export current robots.txt, meta robots, and Bing WMT AI-related settings.
Label each rule: training, grounding/answers, classic search, unknown.
For Google: decide Google-Extended explicitly — do not rely on the shared line until 2027 support ships, and understand the grounding side effect when you do.
For Microsoft: map NoArchive/NoCache pages to training vs grounding intent before March 2027; schedule WMT temporary prefs if needed.
Keep search bots allowed where you want citations unless legal forbids it.
Re-run a fixed set of buyer prompts after any change — citations and mentions lag policy updates by days or weeks.

What we are still waiting on

Final IETF process: Kelley’s draft expires May 3, 2027 unless revised; Content-Usage integration is TBD.
OpenAI conflict rules between shared line and GPTBot.
Anthropic public documentation for the token and any Claude training/search split beyond today’s three bots.

The direction of travel is clearer: training preference becomes standardized; answer visibility stays vendor-specific. Obsurfable’s corpus exists because those two outcomes diverge in production — prompts, answers, brands mentioned, and citations — regardless of what a single robots.txt line says on paper.