What Is llms.txt, and Should You Use It? The Honest 2026 Verdict

What is llms.txt, exactly?

llms.txt is a proposed standard for a markdown file at your site's root that gives large language models a clean, curated guide to your most important content. 

Created by Jeremy Howard of Answer.AI, the llms.txt specification on GitHub defines a simple format: an H1 title, a short description, and a set of markdown links pointing models to your key pages.

The file sits alongside familiar files like robots.txt and sitemap.xml, but it serves a different purpose. Where robots.txt controls access and a sitemap lists URLs, llms.txt is meant to hand a model a concise, human-curated overview it can read without wading through your HTML. 

A minimal one looks like this:

# Your Company

> A one-line description of what you do.

## Docs

- [Getting started](https://example.com/docs/start.md): Setup guide

- [API reference](https://example.com/docs/api.md): Full API

## About

- [Company](https://example.com/about.md): Who we are

‍

That is the whole idea: a short, machine-friendly table of contents. It is elegant, it is cheap to produce, and understanding how AI systems select and read sources is the key to judging whether it earns its place on your site.

What is llms.txt supposed to do?

It is meant to solve a real problem: models struggle to read messy web pages. As the specification explains, context windows are too small to hold most websites in full, and converting cluttered HTML into clean, usable text is difficult and imprecise. A curated markdown file sidesteps that by giving the model exactly the content that matters, in a format it parses easily.

The intended payoff is comprehension. Instead of forcing a model to guess at your structure or trip over navigation, ads and scripts, you hand it a clean brief. In theory, that helps the model understand your site and represent it accurately, which is why so many people assumed it would translate into more citations in AI answers.

The vision goes further than a single file. The proposal imagines companion markdown versions of your pages, so a model following a link from your llms.txt lands on clean text rather than a rendered page full of menus and scripts. 

For documentation especially that is a real convenience, and it is easy to see why the idea caught on so quickly among developers who felt the pain of AI tools misreading their sites.

That assumption is the crux of the whole debate, and it is where intention and reality part ways. The idea is sound. The question is whether the AI systems you care about actually use the file, and that is an empirical question with an increasingly clear answer.

Does llms.txt actually work for AI search?

Mostly no, at least not for getting cited in AI search today. The largest public study to date, Ahrefs' analysis of roughly 137,000 sites, found that around 97% of llms.txt files received no requests at all, and that AI search retrieval bots accounted for barely 1% of the requests the files did get. 

Their blunt conclusion was that for showing up in ChatGPT, Perplexity or AI Overviews, an llms.txt file is largely decoration.

The providers themselves have said as much. Reporting on that data, Search Engine Journal noted Google's position that llms.txt is not used for Search, with Google's John Mueller describing it as a temporary crutch to save tokens for AI coding tools rather than a search signal.

When the platforms you are trying to reach say they do not read the file, the marketing claims deserve real scepticism.

The reason is mechanical, not ideological. AI search crawlers discover content the same way traditional crawlers do, by following links and reading pages, and they do not go hunting for a special file that most sites do not have.

A file the crawler never requests cannot influence what it indexes, no matter how clean the markdown is. That is the gap between how llms.txt was imagined and how the systems actually behave.

This is the honest core of the topic, and it is what most breathless guides leave out. If your goal is to be cited by AI search engines, llms.txt is not the lever.

The crawlers that actually generate your citations are the AI search bots, and they are not the ones fetching these files. That is why we steer clients toward generative engine optimisation fundamentals rather than a file the search systems ignore.

So what is llms.txt actually good for?

It is genuinely useful for one thing: making large documentation sets consumable by AI agents and coding tools. This is not nothing, and it is where the file earns real praise. If you run extensive product docs, an llms.txt gives agents a clean, structured way to navigate and ingest them without choking on rendered HTML.

Cloudflare is a good example of this done well. Its guidance on making documentation consumable by AI describes using llms.txt as a tiered directory of its docs, with a root file linking to product-level files and full-text versions, cutting the tokens a model needs to read them by a large margin. For a documentation-heavy business, that is a concrete, present-day benefit.

The pattern holds across the tools that do read these files. In the Ahrefs data, the requests llms.txt files did attract came disproportionately from coding assistants and auditing tools, not from the AI search engines answering buyer questions.

That tells you exactly who the file serves today: agents and tools pointed directly at your content, working with it deliberately, rather than the discovery systems deciding whether to mention you at all.

The distinction that clears up all the confusion is this: llms.txt helps AI tools that are already trying to read your specific content, such as a coding agent pointed at your docs. It does not help AI search engines decide to cite you in the first place.

Those are different jobs, and the file only does the first one, so pairing that understanding with real answer engine optimisation is how you cover both.

Are there risks to shipping one?

There are two worth knowing, and neither is a dealbreaker, but both deserve attention. The first is maintenance. An llms.txt that drifts out of sync with your real content can point models at stale or wrong pages, which is worse than having no file at all, so an auto-generated, always-current one beats a hand-written file you will forget to update.

Stale files are the more common problem in practice. A business ships an llms.txt during a redesign, moves on, and a year later it still points at pages that have changed, moved or been deleted.

A model trusting that file gets an outdated picture of the business, which is the opposite of what the file was meant to achieve, and no one notices because nothing visibly breaks.

The second is security. Because agents trust the content they ingest, a file that feeds instructions to a model is a surface for prompt injection, where hidden instructions embedded in content get treated as legitimate commands.

If your llms.txt or the pages it links to are auto-generated from user input, review them, because an agent reading a poisoned file can be steered into unintended behaviour.

Neither risk should scare you off, but both argue for the same discipline: keep the file automated, accurate and clean, or do not ship it at all. A neglected llms.txt is a liability, and auditing your generative AI search visibility regularly is how you catch a stale one before it misleads a model.

Should you ship an llms.txt file?

Ship one if it is cheap and you have documentation worth exposing to AI agents; skip it if you are only chasing search citations. That is the honest decision rule, and it depends entirely on why you are considering it.

If your platform or CMS can auto-generate and maintain an llms.txt at no ongoing cost, and especially if you have substantial docs, add one.

It costs you almost nothing and may help agentic tools now and more later. If you run a documentation-heavy product, it is close to a no-brainer.

The calculus is really about opportunity cost. An llms.txt that generates itself costs you nothing and carries a small upside, so it is easy to justify.

A hand-built one that needs constant attention competes for the same hours you could spend on content and structure that demonstrably move citations, and that trade rarely favours the file. Let effort, not hype, decide it.

If you are a typical business hoping llms.txt will get you into ChatGPT and Perplexity answers, do not prioritise it, because the evidence says it will not. Your time is far better spent on the content and technical work that AI search engines actually reward.

There is no harm in a cheap, accurate file, but there is real harm in believing it substitutes for the fundamentals. A quick AI visibility audit will tell you where your effort genuinely pays off.

What should you do instead for AI citations?

Invest in the things AI search engines demonstrably use: genuinely helpful content, clean structure and technical accessibility. Google's own guidance on creating helpful, reliable, people-first content is not a coincidence here; the same qualities that earn search trust earn AI citations, because the systems draw on the same signals of usefulness and expertise.

Concretely, that means writing content built to be quoted, the discipline behind crafting content that gets cited by LLMs, and marking it up so machines read it cleanly, which is where structured data for answer engine optimisation does far more than any llms.txt.

These are the levers with evidence behind them.

None of this is as novel as llms.txt sounds, and that is precisely the point. The work that earns AI citations is the same durable work that earned search visibility: be genuinely useful, be easy to read, be technically reachable. It is less exciting than shipping a new file, but it is what the evidence rewards, and it compounds over time instead of gathering dust.

The businesses that win AI citations are the ones doing that unglamorous groundwork consistently, while their competitors keep chasing the next shortcut.

Your visibility in ChatGPT and Perplexity is won by being useful, credible and readable, not by a file those systems skip.

The same is true for Gemini and every assistant your buyers use. If a past over-investment in shortcuts has left you invisible, the recovery path is the one we map in winning back visibility lost to AI search.

The framing we use at SkyScale is simple: ship an llms.txt if it is free and you have docs, but never mistake it for the real work.

Our case study on rebuilding AI visibility and the broader AI SEO fundamentals show what actually moves citations, and none of it hinges on a file the search engines ignore.