llms.txt: Why the File Barely Affects AI Citations

Andrew Maryasov, AI consultant. In June 2026 I put llms.txt on four of my sites — it took half an hour. Ahrefs had checked the May 2026 server logs across 137,000 domains: 97% of those files didn’t get a single request. Google, in its own guide, called them unnecessary for showing up in AI answers. llms.txt doesn’t make a site more visible to AI, because bots don’t look for it. Visibility comes from crawler access, indexing and content depth.

TL;DR (in 30 seconds)

• 97% of llms.txt files were never requested during May 2026 — Ahrefs, a sample of 137,000 domains with server logs.
• Missing llms.txt files got no requests at all. Bots don’t look for this file; no discovery mechanism exists.
• SE Ranking, across ~300,000 domains, found no link between having the file and how often a site gets cited. When the variable was removed from the model, the model’s accuracy went up.
• Google took both positions in a single week: Search Central writes in its “mythbusting” section that machine-readable files aren’t needed, while Lighthouse adds an llms.txt audit to the “Agentic Browsing” category and turns that category on by default.
• Among the file’s AI readers, search bots trail far behind. The list is topped by GPTBot, a training crawler, and Claude Code, a developer agent; another 12% of requests come from GEO tools that are checking this very file.
• It’s worth half an hour to set up, as a convenience for agents that have already arrived. It’s not worth counting on.

Thirty minutes that changed nothing

On 13 June I set out to roll llms.txt out across four domains: auspex.company, grow2.ai, amaryasov.expert and aelo.cloud. The file is simple — a markdown map of the site for language models. While I was at it, I added a Link header per RFC 8288 with rel="describedby", so a bot would get a pointer to the file in every HTTP response instead of guessing the path. Tidy engineering, thirty minutes of work.

Before deploying I decided to check the obvious: do crawlers reach my sites at all? One command:

curl -sI -A "OAI-SearchBot/1.0" https://amaryasov.expert/

HTTP/2 403. “Your request was blocked.”

Same story with GPTBot, ClaudeBot and PerplexityBot. A regular browser user agent got a clean 200. And yet robots.txt let everyone in — I wrote it myself, and I wrote it to be welcoming. The block sat one level down, in Cloudflare: “Block AI bots” was switched on for this particular domain. It wasn’t on for auspex.company or grow2.ai, which is why I spent six months certain that everything was fine.

So my personal site — the center of the whole entity strategy, the place where the Person schema lives — was completely invisible to ChatGPT, Perplexity, Claude and Gemini. And at that very moment I was carefully leaving a note there for AI.

I fixed it the same day, five minutes in the dashboard. llms.txt shipped the next morning. Then I spent a month reading studies that explained that, of those two actions, only the first one mattered.

What the data shows

Four independent measurements have come out over the past six months. They look at different things — correlation with citations and actual requests in the logs — but they agree on one point.

Study	Sample	Finding
Ahrefs, June 2026	137,000 domains, server logs	97% of files — zero requests in May
SE Ranking, late 2025	~300,000 domains	No significant link to citations
Limy.AI, 90-day log	500+ million AI bot visits	408 requests for llms.txt
Otterly.AI, 90-day server log	62,100+ AI bot visits	84 requests for the file (0.1%)

The details of the Ahrefs study deserve a closer look, because these are request logs, not a survey or a correlation model.

28% of the domains in the sample publish the file. That’s higher than the web average: the sample skews technical, since these are Ahrefs Web Analytics users, meaning people who already do SEO professionally. SE Ranking’s broader sample puts it closer to 10%.

Of the 3% of files that anyone requested at all, 19.5% of the requests came from named AI tools. First on the list is GPTBot, second is Claude Code, and both are ahead of any AI search bot or assistant. Another 12% of requests were generated by GEO and AEO platforms, llms.txt checkers and researchers. To a noticeable degree, the industry is studying itself. The Chrome Lighthouse audit accounted for roughly one request in a thousand.

SE Ranking came at it from the other side and tested the statistical relationship between having the file and citation frequency in AI answers. No relationship turned up, either in the regression or in the XGBoost classifier. More than that: when llms.txt was dropped from the feature list, prediction accuracy improved — the variable was adding noise, not signal.

The mechanism simply isn’t there

Missing llms.txt files didn’t get a single request. Bots don’t show up to probe /llms.txt on the off chance, the way they do with /robots.txt or /sitemap.xml.

Here lies the difference that marketing newsletters work hard to blur. robots.txt works because it’s requested by convention, before the first page. sitemap.xml works because robots.txt points to it and search engines have agreed to read it. Behind llms.txt there is neither a request convention nor an agreement among vendors. None of the big players — OpenAI, Anthropic, Google, Meta, Mistral — has committed to using it as a citation signal.

A file nobody requests can’t influence an answer. This isn’t a “weak effect” that will show up in a bigger sample. It’s a missing channel.

My Link header with rel="describedby" in theory closes exactly this gap: it tells the bot where the file is, no guessing required. The header is on all four domains, it’s served with every response, and none of that has made anyone need the file more. On my own sites I keep llms.txt as a convenience for agents, not as a visibility tactic. The right answer to a question nobody asked is still right, and still unnecessary.

Google took both positions in one week

In May 2026 Google published a guide to optimizing for generative AI features. In a section literally titled “mythbusting,” it says machine-readable files such as llms.txt aren’t needed to appear in generative AI search.

A few days later Lighthouse 13.3 came out and turned on the “Agentic Browsing” category in the default config — the same one that had received an llms.txt presence audit a version earlier.

These are two different consumers, not a mix-up at all. The Search team is describing retrieval: how the system decides which site to pull into an answer. Lighthouse is looking at an agent that has already arrived on the site and has to move around it.

Google’s John Mueller put it this way on the 15 June episode of Search Off the Record: llms.txt gives a model no way to decide which site to show for a specific query. A self-declared file doesn’t tell one site from another — anyone can write that they’re the best source in their niche. Back in 2025 Mueller compared this setup to the keywords meta tag, and the comparison is accurate: an unverified self-description that’s easy to game quickly stops being a signal. The file does have a role, Mueller says, but a narrow one, and it starts after the agent is already on the site.

Who actually needs the file

An unexpectedly useful conclusion falls out of the Ahrefs data. llms.txt does have readers; they just aren’t search bots.

Next to OpenAI’s training crawler GPTBot, the top of the list holds Claude Code — a developer agent. So are Cursor, Copilot, agentic browsers. An agent like that is already on your domain and wants to find the API description without wading through two hundred kilobytes of HTML with navigation, a cookie banner and a footer. A markdown map saves it a step.

Hence the rule I now live by. If your documentation runs past fifty pages, set up llms.txt and llms-full.txt: they have measurable readers. If it’s a corporate site, a blog or a small shop, set it up in half an hour when you have a spare moment, and forget about it; it’s the lowest priority.

What isn’t worth doing at all: pitching this to a client as “SEO for AI.” Easy to sell, impossible to prove, and a year later you’ll have to explain why nothing happened.

What works instead

This is what comes first in any case, and it isn’t an alternative to the file. The order follows the strength of the evidence, not what’s fashionable.

First and crudest — crawler access. I got burned by it, so it goes at the top of the list. A 403 means all the other work is pointless: content, schema, files — none of it exists for someone who never got through the door. Look for the block at the CDN level, not in robots.txt, because that’s where it lives (and, judging by my case, it sometimes switches itself on, bundled with some “recommended” security toggle).

Second — SSR or static HTML. GPTBot, ClaudeBot and PerplexityBot don’t execute JavaScript, so content that gets rendered on the client simply isn’t there for them.

Third — indexing. ChatGPT Search pulls its candidates from a search index; a site outside the index won’t appear in the answer, no matter how many markdown maps you drop in the root.

Fourth — entity consistency: one canonical name everywhere, an @id in the Organization schema, reciprocal sameAs links to Wikidata and LinkedIn. Here I have to ease off on the promises. Schema App, in a controlled study on eleven test pages, got +46% impressions on non-branded queries in 85 days — although two more properties were added alongside sameAs, so the effect can’t be credited to sameAs alone. Ahrefs, on a sample of 75,000 brands, found a 0.66 correlation between mentions and Google AI Overviews, against 0.22 for backlinks. These are correlations on observational data, not proven causation, and I don’t recommend turning them into a promise to a client.

Fifth — content depth, the only part of GEO with peer-reviewed backing. The Princeton GEO paper (KDD 2024) measured nine tactics: statistics in the text give about +41% in passage visibility, direct quotations +37–43%, citing sources roughly +30%. The authors wrote the caveat down themselves: they tested on a Bing Chat proxy and measured the effect only among pages that were already in the top 5. Classic SEO hasn’t “died” here; it’s the prerequisite.

I have numbers of my own too, and they prove nothing. On 14 June I ran twelve prompts through Perplexity: 10 points out of 36 in total, and commercial discovery queries scored a flat zero out of seven. The measurement was taken the day after the deploy, before any reindexing. It’s point A, not a result. I haven’t taken a comparable snapshot with the same method since. In September I measured differently — eighteen prompts through the Claude, OpenAI and Gemini APIs, two runs each — and category-level commercial queries again scored zero on all three engines. The methods differ, so I don’t compare the numbers with each other; only the conclusion matches: on commercial queries, I don’t exist.

Three commands that will tell you more than llms.txt

Run them on your own domain. It takes a minute.

# 1. Does the CDN let AI crawlers in? 403 → the rest of the work is wasted
curl -sI -A "OAI-SearchBot/1.0" https://example.com/ | head -n 1

# 2. Is the content in the raw HTML? Bots don't execute JavaScript
curl -s -A "GPTBot" https://example.com/ | grep -c "<h1"

# 3. What robots.txt actually allows live-retrieval bots to do
curl -s https://example.com/robots.txt

If the first command returned 403, you’ve just found the reason you’re missing from AI answers. That’s exactly how I found it, and it took me six months.

FAQ

Do you need llms.txt to show up in AI answers?

No. Google says outright in the “mythbusting” section of its guide that machine-readable files aren’t needed for that. No major AI vendor has confirmed using llms.txt as a citation signal.

Does llms.txt do any harm?

No, as long as you don’t spend weeks on it. The file blocks nothing and breaks nothing. The only harm: it eats the attention budget that should have gone to crawler access and content.

Then why do big companies set it up?

Mostly because of developer agents. Cursor, Claude Code and agentic browsers read the file by their own tool’s convention. For sites with large documentation it’s a measurable convenience, not a search tactic.

Will this change anytime soon?

It might. Lighthouse already audits the file for agentic browsing. If agent traffic grows and vendors agree on a request convention, the channel will appear. Today it isn’t there.

How is llms.txt different from robots.txt?

robots.txt is requested by convention, before the first page, and it controls access. Nobody requests llms.txt by default, and it controls nothing — it’s a self-description.

What should I do first if the GEO budget is small?

Use curl with an AI user agent to check whether the CDN is returning a 403. It costs zero and closes off the most expensive mistake.

How much does it cost to set up llms.txt?

Half an hour. That’s exactly why the argument around it is so loud: the cost of being wrong is low either way, so both sides can argue forever.

---

Run the three commands above on your own domain. If the first one returned 403, write to me and we’ll work out where the block sits: in Cloudflare, in the WAF or in robots.txt itself.

Related breakdowns: how I evaluate a new AI tool — on separating data from hype, and why I was wrong about AI prices — on revisiting your own claims when the numbers come in.

Written by Andrew Maryasov, AI consultant, founder of Auspex and Grow2.ai. I break down AI agents and automation for business — advisory.

Sources:

• Ahrefs: llms.txt study — 137,000 domains, server logs, May 2026
• Ahrefs: brand mentions and AI Overviews — a sample of 75,000 brands, correlations of 0.664 and 0.218
• SE Ranking: llms.txt — ~300,000 domains
• Limy.AI: llms.txt in 2026 — 90-day log, 500+ million visits
• Otterly.AI: the llms.txt experiment — 62,100+ bot visits, 84 requests for the file
• Search Off the Record, the episode on Markdown and HTML — 15 June 2026, John Mueller’s position
• Google says llms.txt is comparable to the keywords meta tag — Search Engine Journal, a recap of Mueller’s 2025 remark
• Lighthouse v13.3.0 release notes — the “Agentic Browsing” category in the default config
• Google: AI features and your website — the mythbusting section
• Schema App: measurable impact of scaling entity linking — +46% impressions on non-branded queries in 85 days
• GEO: Generative Engine Optimization — Princeton, KDD 2024