llms.txt in 2026: what it does and how to write one

What is llms.txt?

llms.txt is a Markdown file at /llms.txt that gives AI agents a short summary of a site and an edited list of links to its most useful pages. Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed it at llmstxt.org on 3 September 2024, and published a second version in August 2026 based on two years of use.

The problem it addresses is real. Web pages wrap their content in navigation, scripts and ads, and context windows are still too small for most whole sites. An llms.txt file is meant to be what an agent reads first: small enough to load whole, with links to clean Markdown versions of the pages that hold the detail.

It is not a robots.txt for AI: it grants no permissions and blocks nothing. It is not a sitemap, which lists every indexable page; it is a short reading list. And it is not a web standard. The proposal calls itself an informal overview, maintained on GitHub and open for community input.

The llms.txt format, with an annotated example

An llms.txt file is a set of Markdown parts in a fixed order, and only the first is required:

An example llms.txt file with its parts labelled: the H1 with the site's name, a blockquote summary, H2 sections listing links with one-line descriptions, and an Optional section. — The parts of an llms.txt file, on a made-up company.

An H1 with the name of the site or project.
A blockquote (a line starting with >) with a short summary of the key facts.
Optional paragraphs or lists, with no headings, that add detail or explain how to use the links.
Sections under H2 headings, each a Markdown list of links written as - [Page name](full URL): optional notes.
By convention, an H2 called Optional for secondary links an agent can skip. Since version 2 this is only a convention, not an instruction to tools.

The spec's own example is a cut-down file for FastHTML, a Python web library. Here it is part by part:

Part	In the FastHTML file	What it does
H1	# FastHTML	Names the project. The only required line.
Blockquote summary	> FastHTML is a python library which brings together Starlette, Uvicorn, HTMX, and fastcore's FT "FastTags" into a library for creating server-rendered hypermedia applications.	One sentence an agent can rely on if it reads nothing else.
Notes, no headings	Important notes:, then two bullets: not compatible with FastAPI syntax; works with vanilla JS libraries but not React, Vue or Svelte.	Heads off mistakes a model would otherwise make, such as writing FastAPI code.
H2 file list	## Docs, linking the quick start (a .md page) with the note "A brief overview of many FastHTML features", then the HTMX reference.	The core reading list. Links point to Markdown, not HTML.
Second file list	## Examples, linking a Todo list application on GitHub.	A complete app, so an agent can see idiomatic code.
Optional section	## Optional, linking a subset of the Starlette documentation.	Background an agent can skip.

Two details matter more than they look. The links should point to LLM-friendly content: the proposal asks sites to publish a Markdown copy of each useful page at the same URL with .md added (page.html.md) or swapped in (page.md). And the file can live in a folder: /docs/llms.txt covers everything under /docs/, and agents should use the most specific file. Version 2 also recommends rel="describedby" links, as HTML link tags or an HTTP Link header, so agents can find the file without guessing.

What is llms-full.txt?

llms-full.txt is a single file holding the full text of a documentation site in Markdown, rather than links to it. It is not part of the llmstxt.org proposal. Documentation platforms made it common: Mintlify announced automatic llms.txt and llms-full.txt files for the docs it hosts on 20 November 2024.

The appeal is that a developer can paste one URL into an assistant and hand over everything. The catch is size. Anthropic's llms-full.txt for its developer docs was about 50 MB when we fetched it on 5 November 2026, far more than a model can read at once. Version 2 of the proposal goes the other way: a small index, with the detail behind links fetched only when needed. A full file suits a small docs site; a large one is better served by per-page Markdown.

Does llms.txt work? The evidence on rankings and AI citations

No study has found that llms.txt improves rankings or AI citations, and the research we rely on grades it Refuted as a citation lever. The main evidence:

Source	What it looked at	Finding
Google Search Central, guide to generative AI features (updated July 2026)	Google Search, including AI Overviews and AI Mode	Search ignores llms.txt; keeping one "will neither harm nor help" visibility or rankings
Ahrefs, 137,210 domains (June 2026)	Every request to /llms.txt paths in May 2026, from bot analytics	28% published a file; 97% of those files got no requests
SE Ranking, nearly 300,000 domains	How often each domain was cited in sampled LLM answers	No effect; the model was more accurate without the llms.txt variable
Chrome Lighthouse, agentic browsing audits	Whether the file can be fetched	Flags a server error; a missing file (404) is marked not applicable

Google's statement is the clearest. It says you don't need AI text files or Markdown to appear in Search, including its AI features, "as Google Search itself doesn't use them", and that keeping an llms.txt for other services "will neither harm nor help your site's visibility or rankings". If Googlebot fetches the file, that means nothing special: Google crawls many kinds of files without treating them in a special way.

Both studies carry caveats. Ahrefs says its customers skew technical, so 28% adoption is an upper bound, and that a request proves only a fetch, not use. SE Ranking found that 10.13% of its domains had a file, and its results are correlations within one model and dataset.

To be fair to its supporters, none of this shows the file is useless. It shows it is not a search or citation lever, and its author never pitched it as one. The proposal is about agents reading a site while they help a user; as Ahrefs puts it, the AI visibility framing was attached later by the SEO industry.

Which bots actually request llms.txt?

Mostly not AI search bots. In Ahrefs' data, 96% of requests to llms.txt files came from bots, led by SEO audit tools (21.7%), unidentified bots (14.9%), general crawlers such as Googlebot (13.1%) and technology profilers (11.6%). AI bots together made 19.5%:

AI agents and agent infrastructure: 10.5%. Among named AI tools, GPTBot made the most requests and Claude-Code, Anthropic's coding agent, came second.
AI training crawlers such as GPTBot and ClaudeBot: 5.3%.
AI assistants fetching for a user, such as ChatGPT-User and Claude-User: 2.5%.
AI search bots such as OAI-SearchBot and PerplexityBot: 1.1%. Slack's link-preview bot fetched the files more often than PerplexityBot did.

AI bots made no requests at all for llms.txt files that didn't exist, so they don't go looking for one; the 404s came almost entirely from people. Ahrefs' reading is that AI tools fetch the file when a link, an index or a user tells them it exists.

When llms.txt is still worth 15 minutes

llms.txt is worth writing when your readers use coding agents or other AI tools pointed at your documentation, and it costs little anywhere else:

Developer documentation. The proposal says this is where the files are used most, by coding agents looking for API references. OpenAI, Anthropic and Google's Gemini API all publish one for their developer docs.
Coding agents and IDE assistants. Google's John Mueller, quoted by Ahrefs, called the file "not done for search" and a "temporary crutch, perhaps to save some tokens" for AI coding tools reading developer docs.
MCP and tool use. LangChain's open-source mcpdoc, for example, is an MCP server that gives Cursor, Windsurf and Claude Code a list of llms.txt files the user chooses, plus a tool to fetch the pages they link to.
When your platform does it for you. Wix generates a file for every site, and Yoast SEO, AIOSEO, GitBook and Mintlify can generate one. The only work left is checking what it lists.

Otherwise, a marketing site that publishes one loses 15 minutes, and Google says it does no harm. Just don't report it to a client as AI visibility work.

What matters far more for AI visibility than llms.txt

Bot access, server-rendered content, indexing and coverage of the searches engines run decide whether AI engines can find and cite a page:

Search bots allowed. OAI-SearchBot, PerplexityBot and Claude-SearchBot build the indexes ChatGPT, Perplexity and Claude search, and OpenAI says sites that opt out of OAI-SearchBot don't appear in ChatGPT search answers. The AI crawler user-agent list names every bot to check in robots.txt and firewall rules.
Main content in the server HTML. AI fetchers don't run JavaScript, so text rendered only in the browser never reaches them.
Indexed in Google and Bing. Google's AI features use only pages that are indexed and eligible for a snippet, and Copilot grounds its answers in Bing's index.
Coverage of fan-out searches. Engines split a question into sub-searches and cite pages retrieved for those. What moves AI citations grades each factor by its evidence.
Off-site mentions. Being named in the reviews, comparisons and threads engines already cite has strong evidence behind it; how ChatGPT chooses sources shows where those citations come from.

Optimz's free AI crawler checker shows, for one page, which AI bots robots.txt lets in, the line that decides it, whether a firewall challenge answers instead, and whether the content is in the server HTML.

How to write an llms.txt file in 15 minutes

Writing an llms.txt comes down to deciding what the site is, which pages matter and what each one answers. Work in this order:

Write the H1 and a one- or two-sentence blockquote: what the site is, who it is for, and the fact a model most often gets wrong.
Add notes if you need them, with no headings, such as "not compatible with version 1 of the API" or "prices exclude VAT".
Pick the pages that matter, not every page: getting started, pricing, key product pages, policies, API references.
Group them under H2 headings that match how people ask, such as Docs, Pricing and Policies.
Write each item as - [Page title](full URL): one line on what the page answers. Use the final https URL, or the page's Markdown copy if you publish one.
Put nice-to-have pages, such as blog archives and old versions, under ## Optional.
Serve it at /llms.txt as UTF-8 with a 200 status and a text/plain or text/markdown content type.
Test it as the spec suggests: give an assistant only your llms.txt, ask it about your site, and fix what it gets wrong.
Link to it from your docs, footer or README, or with the Link header. Agents fetch the file when something points to it.

A template, one line of the file per bullet. Replace the bracketed text:

# [Site or product name]
> [Name] is [what it is] for [who it is for]. [The fact a model most often gets wrong.]
[Notes: what to read first, units, versions, what you don't do.]
## Docs
- [Getting started](full URL): what a new user does first
- [API reference](full URL): every endpoint, with parameters and errors
## Pricing and policies
- [Pricing](full URL): plans, prices and what each includes
## Optional
- [Blog](full URL): background articles

An llms.txt generator, whether a CMS plugin or an online tool, gets the format right but can't do the editing. Treat its output as a first draft: cut pages nobody asks about and make each note say what the page answers.

Common llms.txt mistakes

The common llms.txt mistakes make the file useless to the few agents that read it, or spend effort in the wrong place:

Blocking the bots you are writing for. robots.txt rules apply to /llms.txt like any other URL, and a firewall that challenges AI user agents serves them a challenge page instead.
Listing URLs that redirect, return 404 or carry noindex. Each redirect is an extra fetch, a 404 is a dead end, and a noindex page is usually one you chose to hide.
Letting it go stale. Generate it from the same source as the site, or add it to the release checklist, so it never lists retired products or old prices.
Serving the wrong thing. Many single-page apps answer any unknown path with the HTML app shell and a 200; Ahrefs had to screen out such soft 404s. The spec names no content type, and the AI labs serve theirs as text/plain; charset=utf-8.
Dumping the sitemap. Hundreds of links with no notes are a sitemap in Markdown.
Leaving it open to edits. Ahrefs warns that agents are built to trust the file and that bad actors already probe it for prompt injection. Keep it in version control.
Selling it as an AI visibility fix. Google says it neither helps nor harms Search, and the citation studies found no effect.

How to check whether AI bots fetch your llms.txt

Search your server or CDN access logs for requests to /llms.txt and look at each user agent:

Filter to the path, for example grep "/llms.txt" access.log, and group hits by user agent.
Sort the bots by job: AI search (OAI-SearchBot, PerplexityBot, Claude-SearchBot), user fetches (ChatGPT-User, Claude-User, Perplexity-User), coding agents such as Claude-Code, training crawlers (GPTBot, ClaudeBot), and the rest, mostly SEO tools and link previews.
Verify IPs before trusting a name. User agents are easy to fake; the AI bot list links each operator's published IP list.
Use CDN logs if a CDN caches the file, because cached hits never reach your origin.
Read the status codes. A 403, 429 or 503 means a firewall or rate limit answered instead. 404s for a file you never published are mostly people, not AI bots.
Remember that a hit means the file was fetched, not that it was used.

Optimz publishes its own file at optimz.com/llms.txt, generated from the same data as the site's pages so it changes when they do. The free AEO audit checks llms.txt alongside AI crawler access, bot challenges and server HTML for up to 10 pages.

Sources