Obsurfable

Why Each AI Engine Reads a Different Search Index — and Why You Need to Monitor All of Them

Obsurfable

When a buyer asks ChatGPT, Claude, and Gemini the same question, they are not searching the same web. They are querying three different search indexes through five different retrieval pipelines — and the sources they cite reflect that architecture, not a unified ranking of the internet.

ChatGPT's web search runs primarily on Microsoft Bing's index, supplemented by OpenAI's own crawler (OAI-SearchBot). Claude routes lookups through Brave Search, which Anthropic lists as a subprocessor and which engineers have confirmed via BraveSearchParams in Claude's web_search tool. Gemini, Google AI Overviews, and AI Mode all ground on Google's own search index — the same crawl, with different citation surfaces on top.

If you optimize for Google Search Console alone, you can be completely absent from Claude and never know why. If you only monitor ChatGPT, you are measuring one retrieval backend out of four. The data on cross-engine overlap is unambiguous: this is not one game with different scoreboards. It is multiple games with different rulebooks.

The four retrieval backends

Only Google and Microsoft own web-scale search indexes. Every other major AI assistant rents retrieval from someone else — or builds a hybrid.

AI productRetrieval backendHow you get in
ChatGPT SearchBing index + OAI-SearchBotAllow OAI-SearchBot; submit via Bing Webmaster Tools and IndexNow
Microsoft CopilotBing indexBing Webmaster Tools and IndexNow
Claude (web search)Brave Search indexStay crawlable; earn Brave ranking (no publisher submit tool)
PerplexityProprietary index (PerplexityBot)Allow PerplexityBot; keep content fresh
Gemini, AI Overviews, AI ModeGoogle Search indexGoogle Search Console, sitemaps (IndexNow does nothing for Google)

This table is the entry fee. Indexing gets you into the pool. What gets cited depends on each engine's reranking, freshness bias, source-type preferences, and answer synthesis — which diverge sharply from there.

ChatGPT: Bing plus OpenAI's own crawl

OpenAI documents three distinct crawlers:

  • OAI-SearchBot — fetches public web content for ChatGPT Search results
  • GPTBot — gathers training data (separate from search retrieval)
  • ChatGPT-User — fetches a specific URL when a user pastes a link

To appear in ChatGPT Search while blocking training, allow OAI-SearchBot and block GPTBot in robots.txt. See Training Crawlers vs Search Crawlers.

Seer Interactive's controlled study found 87% of ChatGPT Search citations match Bing's top results, while only 56% match Google's. If you are not in Bing's index, ChatGPT generally cannot cite you — regardless of your Google ranking.

OpenAI is also building its own curated index. Third-party log studies report OAI-SearchBot crawl volume at roughly triple its mid-2025 level, suggesting OpenAI supplements Bing with direct crawls of high-value sources rather than relying solely on Microsoft's link graph.

Critical technical constraint: most AI crawlers do not execute JavaScript. Client-side rendered content is invisible to OAI-SearchBot. Server-side rendering is mandatory for ChatGPT visibility.

Claude: Brave Search, not Google or Bing

Claude is the engine teams get wrong most often. None of the Google or Bing optimization paths apply.

Anthropic added Brave Search to its subprocessor list in March 2025. Engineers identified a BraveSearchParams parameter inside Claude's web_search tool. Profound's narrow test found 86.7% citation overlap with Brave's top non-sponsored results — tight enough to confirm the partnership.

Your Bing IndexNow setup does nothing for Claude. Your Google Search Console does nothing for Claude. If you rank #1 on Google and are invisible on Brave, you are invisible on Claude.

Claude also uses its own crawlers — Claude-SearchBot for index building and Claude-User for on-demand fetches. Allow Claude-SearchBot in robots.txt alongside Brave discoverability.

Claude's citation profile differs structurally from other engines. SurfacedBy's study of 127,198 citations found Claude sent 0.02% of citations to YouTube and 0.01% to Reddit — compared to 11.2% YouTube and 4.0% Reddit for Google AI Mode. Claude favors documentation, vendor pages, and high factual-density content with verifiable statistics.

Gemini and Google AI: one index, three surfaces

Gemini, AI Overviews, and AI Mode all ground on the Google Search index. There is no separate AI index, no special file to upload, and no AI-specific schema that unlocks citation slots. Google confirmed this in its May 2026 official guide: generative AI features are "rooted in our core Search ranking and quality systems."

But one index does not mean one citation outcome. The three surfaces cite different sources:

  • AI Overviews and AI Mode share only ~13.7% of cited URLs on the same queries (Yext analysis of 6.8M citations)
  • AIO content shifts ~70% of the time when the same query is repeated
  • Gemini chat cites brand-owned websites 52.15% of the time — the highest of any major engine (Yext)

Google's retrieval mechanism for AI answers uses query fan-out: the user's question is decomposed into multiple sub-queries, each run through Google Search, and citations pulled across all sub-results. That is why only 37.9% of AI Overview citations come from the organic top 10 — pages can earn citations by answering sub-questions the headline query never ranked for. See The Gemini 3 Citation Reset.

IndexNow accelerates Bing indexing but does nothing for Google. Google wants sitemaps and Search Console.

Perplexity: its own index

Perplexity operates a proprietary index crawled by PerplexityBot. It is neither Bing nor Google nor Brave. Allow PerplexityBot in robots.txt and prioritize content freshness — Perplexity cites content published within 30 days at an 82% rate.

Perplexity averages 21.87 citations per response (Qwairy, 118,000 responses) — nearly 3× ChatGPT's 7.92 — reflecting a strategy of citing multiple sources per claim rather than selecting a single best source.

The overlap data: why "AI search" is not one channel

Multiple independent studies in 2026 converge on the same finding: cross-engine citation overlap is structurally low.

StudySampleKey finding
CiteMetrix680M citationsOnly 11% of domains cited by both ChatGPT and Perplexity
SurfacedBy127,198 citations, 5 enginesOnly 2.7% of domains (309 of 11,647) cited by all five
SurfacedBySame sample69.6% of domains cited by exactly one engine
BrightEdgeCross-engine analysisAs little as 16% overlap between citation sources on any two engines
Foglift1,373 AI answers, 5 enginesChatGPT ↔ Claude citation overlap: 0.027 Jaccard
Yext6.8M citationsAI Overviews ↔ AI Mode share only 13.7% of URLs

Read the SurfacedBy number again: 2.7%. Of 11,647 domains cited at least once across ChatGPT, Claude, Gemini, Perplexity, and Google AI Mode, only 309 appeared in all five. Add domains cited by four engines and you still reach only ~7%.

Winning on one engine barely carries to the others.

CiteLens: same query, different organic pools

A July 2026 CiteLens study ran 320 buyer queries through ChatGPT, Perplexity, Claude, and Google AI Mode, then checked where every cited source ranked in Google and Bing organic results.

EngineShare of citations from Google top-10
Google AI Mode93%
Perplexity89%
Claude53%
ChatGPT30%

ChatGPT pulls 70% of its cited sources from pages that rank in neither Google's nor Bing's top 10. Those sources come from Wikipedia (~48% of ChatGPT citations in some analyses), direct brand mentions, entity recognition signals, and OpenAI's supplemental crawl — not from traditional SERP position.

This is not a quality judgment. It is an architectural fact. ChatGPT's retrieval stack weights different signals than Google's, because it reads from a different index with different ranking mechanics.

What this means for your strategy

1. Indexing is platform-specific.

Getting into Google's index does not get you into Bing's. Getting into Bing does not get you into Brave's. Each backend has its own crawl, indexing pipeline, and publisher tools. Audit robots.txt for every retrieval bot: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot.

2. Optimization must be engine-aware.

A YouTube strategy that wins Google AI Mode (11.2% of citations) is nearly irrelevant for Claude (0.02%). Reddit drives 46.7% of Perplexity's top citation share but only 5% of Gemini's — a 9× gap between products from the same company (CiteMetrix). Match source type to engine, not to a generic "AI search" playbook.

3. Monitoring one engine is monitoring one backend.

If you only check ChatGPT, you see Bing-weighted retrieval. If you only check Gemini, you see Google's index. Neither tells you what Claude (Brave) or Perplexity (proprietary) are doing with the same buyer questions.

4. Parallel observation is the minimum viable program.

Run the same prompt set across ChatGPT, Claude, Gemini, Perplexity, and Google AI surfaces on a repeated schedule. Compare:

  • Citation presence (is your domain in the source list?)
  • Brand mention (is your name in the answer text?)
  • Competitor positioning (who gets named and recommended?)
  • Source persistence (does the citation hold across runs?)

Single-engine monitoring creates blind spots that look like strategy when they are actually incomplete measurement.

The practical playbook

Step 1: Map your buyers to engines

Users under 44 average five search platforms (EMARKETER, 2026). ChatGPT processes 72 billion messages per month; Google still processes 417 billion searches. AI search is supplementary for most categories today — but the engine mix varies by audience, vertical, and query type.

Identify which engines your buyers actually use. A B2B SaaS buyer may research on ChatGPT and Perplexity. A local service buyer may hit Google AI Mode. A developer may prefer Claude. Do not assume one engine represents your market.

Step 2: Verify index presence per backend

BackendVerification tool
GoogleSearch Console → URL Inspection
BingBing Webmaster Tools → URL Inspection
BraveNo publisher tool — check via site:yourdomain.com on search.brave.com
OpenAI crawlCheck server logs for OAI-SearchBot; allow in robots.txt
PerplexityCheck server logs for PerplexityBot; allow in robots.txt

Step 3: Run parallel prompt monitoring

Define 15–30 buyer-intent prompts. Run each against all relevant engines weekly (minimum). Track citation rate, mention rate, and competitor share per engine — not aggregated.

A brand with 80% citation presence on Gemini and 5% on ChatGPT does not have "42.5% AI visibility." It has a Gemini strength and a ChatGPT gap. Aggregating hides the gap.

Step 4: Build engine-specific content distribution

EngineHighest-leverage surfaces
ChatGPTBing-indexed pages, Wikipedia, major publishers, entity-rich brand mentions
ClaudeDocumentation, vendor pages, verifiable statistics, Brave-indexed content
Gemini / Google AIYouTube, structured brand pages, topic clusters for fan-out sub-queries
PerplexityFresh content (<30 days), Reddit, G2, niche review sites

Earned media drives 84% of AI citations across platforms (Muck Rack, 25M links) — but the specific third-party surfaces each engine trusts differ. See Only 2.7% of Domains Are Cited by All Five AI Engines.

Step 5: Treat model updates as re-indexing events

Each model upgrade can shift citation patterns independently of your content. GPT-5.5 redirected 96% of ChatGPT citations to third-party sites. Gemini 3 collapsed top-10 overlap from 76% to 38%. When a major model launches, rerun your full parallel prompt audit — not just the engine where you are strongest.

Why parallel monitoring is Obsurfable's core thesis

Traditional SEO tools report one index — Google's. Bing Webmaster Tools now offers AI Performance reporting for Microsoft's ecosystem. But no single platform shows you what Claude (Brave), Perplexity (proprietary), and ChatGPT (Bing + OAI-SearchBot) are saying about your brand on the same prompts.

Obsurfable runs your defined prompts across AI search engines in parallel and tracks brand mentions, competitor positioning, and cited sources over time — with repetition built in. When ChatGPT cites you via Bing but Claude does not find you on Brave, you see both outcomes side by side, not blended into a single score.

The Visibility Director flags engine-specific gaps — "you appear on Gemini but not ChatGPT for this prompt" — and drafts content designed to close them on the relevant retrieval backend.

This is not a nice-to-have for enterprise brands. With 2.7% five-engine domain overlap, single-engine monitoring is structurally incomplete. The question is not whether to monitor in parallel. It is whether you discover the gaps yourself or your competitors exploit them first.

For platform-specific playbooks, see ChatGPT vs Perplexity: Separate AEO Strategies. For crawler configuration, see Training Crawlers vs Search Crawlers.

FAQ

If I rank #1 on Google, will I appear in ChatGPT?

Not necessarily. Only 56% of ChatGPT citations match Google's top results, and ChatGPT pulls 70% of sources from outside both Google and Bing top-10. Google ranking is necessary for Gemini visibility but insufficient for ChatGPT.

Does Claude use Google at all?

No. Claude's web search retrieves from Brave Search's independent index. Google Search Console and Google SEO have no direct effect on Claude citations.

Is Perplexity on Bing or Google?

Neither. Perplexity operates its own index via PerplexityBot. It requires separate crawl permission and a freshness-first content strategy.

Can I use IndexNow for all engines?

IndexNow helps Bing-backed products (ChatGPT, Copilot) but does nothing for Google or Brave. There is no universal indexing accelerator across all four backends.

How often should I run parallel checks?

Weekly minimum for priority prompts. Citation sets rotate rapidly — Google exchanges 56% of AI Mode sources weekly; ChatGPT exchanges 74%. See AI Citation Volatility.

Bottom line

ChatGPT reads Bing. Claude reads Brave. Gemini reads Google. Perplexity reads its own index. Only 2.7% of cited domains appear across all five. Optimizing for one backend is optimizing for one retrieval pipeline. Monitoring one engine is measuring a fraction of your buyers' discovery paths. Parallel observation across engines is not a premium feature — it is the minimum viable AEO program.