What is llms.txt?

Google’s AI optimisation guide says plainly that Google Search “itself doesn’t use” llms.txt or other AI text files, so publishing one won’t help you rank or appear in AI Overviews or AI Mode. Google adds that it is “completely fine” to maintain one for other systems that do use it. Jeremy Howard of Answer.AI proposed the file in September 2024 to help AI tools use a website at inference time — while answering a question — rather than for training.

What does the evidence say?

• Published widely, read rarely. Ahrefs examined 137K sites that use its analytics in May 2026: 28% had an llms.txt, yet 97% of those files received no requests at all that month.
• AI search bots barely look. AI retrieval bots made just 1.1% of the requests that did arrive; OAI-SearchBot and PerplexityBot barely registered, and AI bots never went looking for files that didn’t exist.
• Coding agents do. Anthropic’s coding agent, Claude Code, out-fetched every AI retrieval bot, assistant and training crawler except GPTBot and one indexing service. That fits the proposal’s own example of a coding agent fetching a library’s documentation.
• Tools now check for it. Chrome’s Lighthouse added agentic browsing audits in May 2026, including an llms.txt presence check. That is an agent-readiness hint, not a Search ranking signal.

How does it work?

The file is plain Markdown. An H1 with the site or project name is the only required part. After it can come a short summary in a blockquote, optional notes, and H2 sections listing links with one-line descriptions; a section titled “Optional” marks links an agent can skip. The proposal also suggests clean Markdown copies of key pages (Markdown for agents). Unlike robots.txt, it grants or blocks nothing, and no AI system is obliged to read it.

# Example Billing API

> REST API for GST-compliant invoices. Base URL: https://api.example.com/v2

## Docs
- [Quickstart](https://www.example.com/docs/quickstart.md): API keys and a first invoice
- [Invoices API](https://www.example.com/docs/invoices.md): endpoints and fields

## Optional
- [Changelog](https://www.example.com/changelog.md)

How to get it right

1. Decide whether you need one. It is worth doing for software, API, SaaS and documentation-heavy sites whose users work with coding agents, and a low priority for a clinic, shop or restaurant.
2. Generate it automatically from your CMS or build so it never goes stale. We publish one for this site, generated at build time, because it costs nothing to keep accurate — not because we expect it to move rankings.
3. Link canonical, indexable pages with one-line descriptions, and keep the file short.
4. Treat it as security-sensitive. Ahrefs warns that agents are built to trust the file and that potential bad actors already probe llms.txt for prompt injection. Restrict who can edit it and keep it in version control.
5. Check your server logs after a month to see who actually fetches it.

Common mistakes

• Expecting rankings or AI citations. Google Search ignores it, and AI search bots rarely request it.
• Using it to block or license content. It is not access control; use robots.txt or content signals for that.
• Doing it before the basics. Crawlable HTML, a clean sitemap and an accurate Business Profile matter far more.
• Letting it drift. In Ahrefs’ words, “a stale or compromised file misleads every agent that reads it”.

In practice

A Kochi software company with public API docs has a real reason to publish one: developers using coding assistants can pull accurate, current documentation into their tools. A Chennai restaurant can skip it and spend the hour on its Business Profile and menu page. Our llms.txt implementation guide covers the setup steps.