llms.txt After Eighteen Months: What the Request Logs Show

I sat down to add an llms.txt to this site in twenty minutes and spent the afternoon reading crawler studies instead. Our robots.txt has let every AI crawler in since the day the site launched, the content is already markdown underneath, and generating a curated index of the tools and articles would have been trivial. What stopped me was a question I should ask more often before shipping a file that only machines read: who requests it?

The answer, eighteen months after the spec appeared, is almost nobody — and yet I would still ship one, for a reason that has nothing to do with search. Both halves of that are worth explaining, because the SEO conversation about this file has almost no data in it.

The File Itself

llms.txt is a markdown document at your site root, proposed by Jeremy Howard in late 2024. Not a directive format like robots.txt, not an XML index like a sitemap — a hand-curated page of prose and links, meant to give a model a clean map of what matters on your site:

# Rich Dev Tools

> Browser-based developer tools and articles: identifiers, encoders, formatters,
> SEO and security utilities. Everything runs client-side, nothing is uploaded.

## Tools
- [UUID generator](https://richdevtools.com/generators/uuid): UUID v1 through v7, in-browser
- [JWT decoder](https://richdevtools.com/security-tools/jwt): decode and sign, client-side only

## Articles
- [What your JSON parser does when the spec says nothing](https://...): cross-runtime behaviour

## Optional
- [About](https://richdevtools.com/about)

The convention has two variants: llms.txt as the curated index, and llms-full.txt as the entire content of your documentation concatenated into one file for models that would rather ingest everything than follow links. Where robots.txt says what a crawler may fetch and a sitemap says what exists, llms.txt says what you think matters. That is the pitch, and it is a good one.

Adoption Went Up. Requests Did Not.

Here is where the story gets uncomfortable for anyone selling llms.txt as an SEO tactic.

SE Ranking's May 2026 study of roughly 300,000 indexed domains found 10.13% had an llms.txt, up from 0.4% in April 2025. A twenty-five-fold rise in thirteen months looks like a standard being adopted. Then look at the same study's traffic breakdown: 0% of the top 1,000 domains by traffic publish one. The adoption is concentrated in small and mid-sized sites, which is the signature of a tactic spreading through SEO advice rather than through engineering necessity.

The request side is worse. Across a 90-day window covering more than 500 million AI bot visits, 408 requests targeted llms.txt. A separate, smaller measurement found 84 requests out of 62,100 AI bot visits — 0.1%. Whichever number you prefer, the crawlers doing the actual work of building AI answers are not asking for the file.

That last row is the one that ended my afternoon. SE Ranking modelled citation likelihood and found that removing the llms.txt variable improved their model's accuracy — the file was adding noise, not signal. Publishing it did not make sites more likely to be cited once you account for the things that actually correlate.

Google has been blunt about why. Gary Illyes said in July 2025 that Google does not support llms.txt and has no plans to, and John Mueller compared it to the keywords meta tag. That comparison is sharper than it sounds: the keywords tag died because a self-declaration nobody can verify is an invitation to lie, and a search engine that trusts it gets gamed within a week. llms.txt has exactly that shape — a file where a site describes its own importance, with no cost to exaggeration.

The Audience That Does Read It

So why would I still ship one? Because the file has quietly found a different job.

Coding agents read it. Cursor, Continue and Cline point at documentation URLs, and MCP documentation servers use the same convention to decide what to pull into context. If your product has a developer-facing API or library, an llms.txt — and especially an llms-full.txt — is the difference between an agent reading your actual docs and hallucinating your API from a two-year-old training snapshot. That is a support-cost argument, not a ranking one.

Some engines say they use it. Perplexity has said it retrieves the file to help prioritise pages, and Anthropic publishes one on its own site. Neither is a documented retrieval guarantee, and I would not build a strategy on it, but it is not zero.

It is a forcing function. Writing a curated index of your best pages, in your own words, with a one-line description of each, is an exercise that exposes how much of your site is filler. I found three tool pages whose value I could not describe in a sentence. That was worth the afternoon on its own.

None of those reasons is "it will get you cited in AI Overviews". If someone sells you the file on that basis, ask them for the request logs.

If You Ship One, Ship It Properly

Twenty minutes, once, no maintenance burden:

• Curate, do not dump. The point is the subset. If your llms.txt lists every URL, you have written a worse sitemap. Twenty to fifty entries for a typical site.
• One line of description per link, written for a reader who has never seen your site. This is the part models can actually use.
• Absolute URLs, canonical ones. No tracking parameters, no redirects.
• Plain markdown. It is parsed as text, so keep the structure boring: # title, blockquote summary, ## sections, link lists. If you generate it from your CMS, check the output renders as valid markdown rather than assuming it does.
• llms-full.txt only if you have documentation worth ingesting whole. For a marketing site it is dead weight; for an API reference it is the whole point.
• Do not block it. An llms.txt behind a bot-mitigation rule that returns a challenge page to unknown user agents is a file nobody can read — the same failure mode that breaks social link previews.

Then leave it alone. It is not a channel to optimise; it is a business card.

What Actually Moves Citations

If the twenty minutes were meant to buy AI visibility, spend them here instead, in this order:

1. Let the right crawlers in. GPTBot, ClaudeBot, PerplexityBot and friends need to be explicitly allowed, and a blanket bot rule blocking them is the most common own goal I find. This costs nothing and is the only item on the list with a hard on/off switch.
2. Serve content in the HTML. Retrieval crawlers do not run your JavaScript. If the answer only exists after hydration, it does not exist.
3. Make passages quotable. Self-contained claims, numbers, dates, tables — the structural work that gets you cited is the same work that makes a page useful to a human skimming it.
4. Ship the metadata you can verify. Canonical, title, description, and structured data that matches the page — a meta tag audit and a valid JSON-LD block are checkable by a machine in a way that a self-declared index never will be.
5. Keep it current. Recency is doing real work in answer engines, and it is one of the few signals where a small site can beat a large one, as I argued in SEO for AI search.

Ship It for the Agents, Not for Google

My verdict after the afternoon: publish an llms.txt if developers or coding agents consume your documentation, because there the file has a demonstrable reader and a concrete payoff. Publish it for AI search visibility and you are shipping a file that 408 requests in 500 million visits suggests nobody fetches, on the recommendation of people who have not looked. The honest version of the advice fits in one sentence — it costs twenty minutes, it will not hurt you, and anyone promising it will lift your citations is describing a mechanism that does not currently exist.

Sources: SE Ranking's llms.txt study and Search Engine Journal's coverage of the 300k-domain analysis.