Obsurfablellms.txt is a plain-text file you place at the root of your domain (yoursite.com/llms.txt) that gives AI systems a curated, Markdown-formatted map of your most important content. The idea is simple: instead of forcing a model to crawl and parse your entire site, you hand it a clean summary and links to the pages that matter.
That is the pitch. The reality in 2026 is more complicated, and the honest answer to "do you need one" is: probably not for AI search visibility, but quite possibly yes if you publish developer documentation.
This article explains what the file is, where the idea came from, what the major AI platforms actually do with it today, and how to decide whether it deserves a place on your roadmap.
What is llms.txt, exactly?
The llms.txt convention was proposed by Jeremy Howard, co-founder of Answer.AI and fast.ai, in 2024. The concept borrows from robots.txt and sitemap.xml but serves a different purpose.
robots.txttells crawlers what they are allowed to access.sitemap.xmllists every URL you want indexed.llms.txtis meant to be a curated, human-readable summary: a short description of what your site or product is, followed by a hand-picked list of the most useful links, often with a one-line note on each.
A minimal example looks like this:
# Obsurfable
> Obsurfable helps companies monitor and improve how they appear in AI-generated answers.
## Docs
- [Getting started](https://obsurfable.com/docs/getting-started): set up your first prompt set
- [Insights](https://obsurfable.com/docs/insights): turn retrieval results into recommendations
## Key pages
- [What is AEO?](https://obsurfable.com/resources/articles/what-is-answer-engine-optimization-a-clear-guide-to-aeo)
Some sites also publish an llms-full.txt, which is a single-file export of an entire documentation set in Markdown, designed to be dropped straight into a model's context window.
The stated goal, in Howard's original proposal, is to help LLMs use a site "at inference time" — a clean entry point so an assistant or coding agent can orient itself without crawling everything.
Does anyone actually read it?
This is where the marketing and the evidence diverge sharply.
The SEO industry attached an "AI visibility" promise to llms.txt as adoption spread — the speculation being that if you published one, ChatGPT, Perplexity, Gemini, and Google AI Overviews would reward you with more citations. That promise has not held up.
Here is what the major platforms say and do as of mid-2026:
| Platform | Official position on llms.txt |
|---|---|
| On the record that it does not use llms.txt for Search or AI features. John Mueller compared it to the deprecated keywords meta tag; Gary Illyes confirmed no support and no plans. Google's AI guidance states the file "won't negatively or positively impact your visibility or rankings." | |
| OpenAI | Crawler documentation tells site owners to control GPTBot, OAI-SearchBot, and ChatGPT-User via robots.txt. The word "llms.txt" does not appear. OpenAI does publish an llms.txt for its own developer docs. |
| Anthropic | Same pattern: guidance points to robots.txt for crawler control. Anthropic publishes llms.txt and llms-full.txt at its docs site — a developer-tooling use, not a confirmed citation signal. |
| Perplexity | Has been the most receptive historically, but has not committed to it as a ranking or citation factor. |
The independent data reinforces this. An Ahrefs analysis of roughly 137,000 sites found that 97% of llms.txt files were never read by an AI crawler at all. Adoption is growing fast — Originality.ai documented an 8.8x increase — but growth in publishing the file is not the same as growth in anything consuming it.
There is one telling exception. In log studies, the bot fetching llms.txt most aggressively is not a search or training crawler — it is Claude-Code, Anthropic's coding agent. That is a strong hint about where the file's real value lives.
The one thing Google did that muddies the water
In late May 2026, Google managed to take both sides of the argument in under a week. Its guidance on optimizing for AI features told site owners that machine-readable files like llms.txt were unnecessary "mythbusting." Days later, the Chrome team shipped an experimental llms.txt check inside Lighthouse's Agentic Browsing audits, with documentation noting that without the file, agents may spend more time crawling a site to understand its structure.
When pressed on the contradiction, Mueller clarified that llms.txt is "not done for search." He called it "a temporary crutch, perhaps to save some tokens" for AI coding tools parsing developer documentation — not something a typical content site needs to worry about.
That framing is the key to the whole debate.
So who should publish one?
The distinction that matters is search citation vs. agent consumption.
You probably do NOT need llms.txt if:
- Your goal is to appear more often in ChatGPT, Gemini, or Google AI Overviews answers. There is no evidence it helps, and the platforms that drive most AI search traffic say they do not use it.
- You run a marketing site, blog, or ecommerce store. Your visibility comes from crawlable, well-structured content and being referenced across the web — not from a file the answer engines ignore.
You probably SHOULD publish one if:
- You have developer documentation (an API, SDK, or framework) that developers load into coding assistants like Cursor, Claude Code, or other IDE-based agents. When a user points an agent at your docs, a clean
llms.txtorllms-full.txtlets it orient quickly and generate more accurate code against your product. This is exactly why OpenAI, Anthropic, and Perplexity publish their own. - You want to future-proof cheaply. Google has been vocal that the future of search is agentic. If agents end up mediating AI search rather than retrieval bots fetching pages directly,
llms.txtcould start to matter through the agent layer. Publishing a good one is low-effort insurance — just don't expect it to move citations today.
If you do publish one, do it right
Publishing is only half the job. Agents fetch llms.txt when directed to it, not speculatively — so an orphaned file nobody links to is unlikely to be picked up. A few practical notes:
- Keep it curated, not exhaustive. This is not a sitemap. Link to your best, most authoritative pages with a short description of each.
- Lead with a clear brand summary. One or two sentences on what you are and who you serve. This is the kind of entity-level clarity that helps any AI system describe you correctly.
- Make sure it returns cleanly. The Lighthouse audit flags server errors on a broken
llms.txt. If you publish one, make sure it resolves without errors. - Reference it where agents will look. Link it from your docs and, if relevant, mention it in your
robots.txt. - Don't treat it as a substitute for crawlable content. A company publishing
llms.txtfor its docs is not the same as that file getting your pages cited. The fundamentals of answer engine optimization still do the heavy lifting.
What actually drives AI visibility instead
If llms.txt is not the lever, what is? The 2026 evidence points consistently to the same things:
- Crawlable, extractable content that answers real questions clearly and early on the page.
- Presence on sources models already trust — Reddit, YouTube, review sites, and reputable editorial coverage.
- Entity clarity so systems understand what you do and can describe you accurately.
- Being the clearest answer to specific questions, since most AI citations go to the long tail rather than a handful of giant domains.
None of that requires a .txt file at your root. It requires content and structure that answer engines can actually use.
How Obsurfable fits
The reason the llms.txt debate persists is that most teams cannot see whether anything they do moves the needle. They publish a file, or restructure a page, and then guess.
That is the gap Obsurfable closes. You define the Prompts you care about, run retrieval to see how ChatGPT and other models actually answer them, and check whether you are mentioned or cited. Insights turn those results into concrete recommendations. Instead of arguing about whether llms.txt helps in the abstract, you can watch your actual citation share over time and invest in the things that demonstrably move it.
FAQ: llms.txt
Is llms.txt a ranking factor?
No. No major AI platform treats it as a ranking or citation factor, and Google has explicitly stated it does not affect Search visibility.
Do ChatGPT and Gemini read my llms.txt?
There is no confirmation that they do, and OpenAI's and Google's own documentation points to robots.txt for crawler control. Log studies show near-zero fetch rates from mainstream AI crawlers.
What is the difference between llms.txt and robots.txt?
robots.txt controls which crawlers can access your site. llms.txt is a curated content summary meant to help LLMs and agents understand your site. They solve different problems and are not interchangeable.
What is llms-full.txt?
A single-file Markdown export of an entire documentation set, designed to be loaded directly into a model's context window — most useful for developer docs consumed by coding agents.
Will llms.txt matter more in the future?
Possibly. If AI search becomes agent-mediated, the file could gain influence through the agent layer. It is cheap insurance to publish a good one, but as of 2026 it does not measurably affect AI search citations.
The bottom line
llms.txt is a genuinely useful idea for one specific job: helping coding agents and documentation tools consume developer docs efficiently. That is why the AI labs publish it for their own docs. But the broader promise — that adding an llms.txt will get your brand cited more often in AI answers — is not supported by how the major platforms behave in 2026.
If you have developer docs, publish a clean one. Otherwise, spend the time on content that answer engines actually read, and measure whether it works.