llms.txt: What It Does, What It Doesn't, and Whether Your Site Needs One
What llms.txt is, what Google says about it, where it is genuinely used (developer docs and coding agents) and how to decide if your site needs one.
Quick answer
llms.txt is a proposed Markdown file at the root of a site that summarizes it and lists its most useful pages for language models. It is not a standard, it controls no access, and Google states that Google Search ignores it. Its real use today is in developer documentation, where coding agents and documentation tools read it to find the right pages. Add one if you publish technical docs or an API; for a typical business or ecommerce site it is optional and should never replace crawlable pages, good content and correct robots.txt rules.
What the proposal specifies
Jeremy Howard of Answer.AI proposed llms.txt in September 2024, at llmstxt.org. The idea: websites are built for people and browsers, with navigation, scripts and layout that waste a model's limited context. A small Markdown file at `/llms.txt` can point a model straight to the content that matters.
The proposal describes a simple structure: an H1 with the site or project name, a short blockquote summary, optional notes, then sections of Markdown links with one-line descriptions. A section titled "Optional" marks links that can be skipped when context is short. It also suggests offering Markdown versions of pages (for example `page.html.md`) and a fuller `llms-full.txt` with content inlined.
# Example Analytics
> Example Analytics is a privacy-focused web analytics product with a JavaScript tracker, a REST API and a self-hosted option.
## Docs
- [Quickstart](https://example.com/docs/quickstart.md): install the tracker and see your first report
- [REST API reference](https://example.com/docs/api.md): authentication, endpoints, rate limits
- [Self-hosting](https://example.com/docs/self-hosting.md): requirements and upgrade process
## Optional
- [Changelog](https://example.com/changelog.md)What Google and other platforms say
Google is the only major search platform to address llms.txt directly. Its guide to optimizing for generative AI features (May 2026, updated July 2026) says Google Search ignores llms.txt files, that you do not need special machine-readable files to appear in Search or its AI features, and that creating one will neither harm nor help your visibility. Google's earlier AI features documentation makes the same point about AI text files in general.
OpenAI and Anthropic document their crawlers and robots.txt controls but do not say their search products use llms.txt as a ranking or citation signal. Independent log studies published in 2026 found that most llms.txt files on the web receive few or no requests from AI crawlers. The honest summary: there is no evidence that llms.txt improves AI search visibility.
Key takeaway
If someone sells llms.txt as an AI search ranking tactic, ask for the platform documentation that supports it. As of October 2026, none of the major platforms provide it.
Where llms.txt is genuinely useful
The file earns its place where a model is actively looking for documentation on someone's behalf. Coding agents and IDE assistants often need to read a library's or API's documentation in the middle of a task, and a curated index with Markdown versions saves them from scraping navigation-heavy HTML. Many documentation platforms now generate llms.txt automatically, and developer-focused companies publish one for their docs. Anthropic's Claude Code documentation, for example, links to its own llms.txt index from every page.
It is also a convenient way to give your own AI tools (an internal assistant, a support bot, an MCP server that exposes your docs) a single, maintained entry point.
| Site type | Recommendation |
|---|---|
| API or developer documentation | Yes: llms.txt plus Markdown versions of key pages |
| SaaS with substantial help docs | Useful for help-center content; keep it generated from the docs source |
| Business or service website | Optional; low effort, low expected impact |
| Ecommerce store | Low priority; focus on product feeds, structured data and crawlable product pages |
| News or publisher site | Low priority; crawler policy matters more |
llms.txt vs robots.txt vs sitemap.xml
The three files are often confused. Only robots.txt has an access meaning, and only robots.txt and sitemaps are used by search engines.
| File | Purpose | Used by | Standard? |
|---|---|---|---|
| robots.txt | Tell crawlers which paths they may request | Search engines and AI crawlers | Yes (RFC 9309) |
| sitemap.xml | List URLs for discovery and recrawling | Search engines | Yes (sitemaps.org protocol) |
| llms.txt | Curated reading guide for language models | Some developer tools and AI assistants | No, a community proposal |
If you add one, do it properly
- Generate it from your content source (docs repo or CMS) so it never goes stale
- Link to canonical, publicly accessible URLs only
- Provide clean Markdown versions for the pages you list
- Keep descriptions factual: what each page answers
- List only content you are happy for anyone to extract
- Keep robots.txt, sitemaps and on-page content as the primary signals
Publishing developer documentation?
ZSpace Labs builds documentation sites and developer portals that work for people, search engines and coding agents, including generated llms.txt and Markdown endpoints. See website development.
Conclusion
llms.txt is a reasonable idea with a narrow, real use: helping AI tools navigate documentation. It is not a search ranking factor, Google ignores it, and it does not control access. Add it where tools will read it, generate it automatically, and spend your AI search effort on the things platforms document, starting with crawler access and content worth citing.
Common questions.
A proposed convention, published by Jeremy Howard in September 2024, for a Markdown file at /llms.txt that gives language models a short summary of a site and a curated list of links to the most useful pages, ideally with clean Markdown versions.