llms.txt: What It Is and How It Helps AI Search Find Your Site

llms.txt is a plain-text Markdown file, normally served at /llms.txt on your domain, that tells AI systems what your site is and which of its pages are worth reading. It was proposed in September 2024 by Jeremy Howard, the AI researcher behind fast.ai, and the specification lives at llmstxt.org. Answer engines with search and coding agents are the readers it is written for; Google Search itself does not use it. Here is what the file is, how to write one, and what it can and cannot do for your visibility in AI answers.

What llms.txt is, and the problem it solves

HTML pages are built for people. A typical page wraps its content in navigation, ads and JavaScript, and converting it back into clean text is “difficult and imprecise,” as the llms.txt background puts it. Language models pay for that chrome in tokens and attention, and a context window is too small for an entire website anyway.

llms.txt is the fix: a small, curated file that tells a model what the site is and points it at the pages worth reading, in plain Markdown. The v2 specification, published in August 2026, names exactly these parts, in this order:

1. An H1 with the name of the site (the only required element).
2. A blockquote with a short, factual summary.
3. Optional detail sections without headings.
4. One or more H2 sections holding “file lists”: Markdown links to the pages an agent should read, each with a note about what it contains.

The file stays small enough to fit in a context window, and the detail lives behind the links: an agent reads llms.txt, then fetches only the pages it needs. The format is “human and LLM readable, but also in a precise format allowing fixed processing methods” (llmstxt.org).

It is a proposal, not a formal standard, and it is meant to be served at /llms.txt or at a subpath such as /docs/llms.txt to cover only that section of a site.

Who reads llms.txt

The most eager readers today are software agents, not search engines:

• Coding agents. “Coding agents and document tools are already reading llms.txt files to help speed up API lookups,” Search Engine Journal reports.
• Documentation platforms. Mintlify generates an llms.txt for every site it hosts, one reason thousands of sites now publish the file.
• The AI labs themselves. OpenAI, Anthropic and Google’s Gemini team publish llms.txt files for their own developer docs, llmstxt.org notes. Working examples live at Cloudflare’s docs and on OpenAI’s developer site.

Answer engines are the audience the proposal was written for. The spec’s background describes a chat assistant with search reading pages to answer questions about a product, and CMSWire names ChatGPT, Perplexity and Google’s AI Overviews as the discovery layer reshaping the buyer journey. The same article warns that platform support is “inconsistent” and that adoption, in one estimate, still sits under 10%.

The honest version: llms.txt is a legibility play, not a ranking lever. Google’s Search team has said for over a year that supporting llms.txt is not on its agenda, and Google’s AI optimization guide states that the file “will neither harm nor help your site’s visibility or rankings,” per Search Engine Journal. You are writing for agents and answer engines, and that is fine: those are the channels where quoted traffic is growing fastest.

How llms.txt differs from robots.txt and sitemap.xml

robots.txt is a gate: it tells crawlers where they are not allowed. sitemap.xml is an inventory: it tells a crawler every page it should consider. llms.txt is a briefing: it tells a thinking model what you are and where to look first. They work together. You still want a sitemap so Googlebot discovers your pages, and you still want robots.txt to keep unwanted crawlers out. llms.txt sits on top and points the agents that read it at your best material, which is why a good llms.txt lists a handful of high-value pages rather than everything you have ever published.

If you want all three compared in a walkthrough, Ahrefs’ AEO course lesson “Technical SEO for AI” shows how robots.txt, GPTBot and llms.txt each affect whether AI systems can read your site:

Writing the intro paragraph: concrete facts, not slogans

The blockquote is the only place your llms.txt makes claims, and an agent may quote it almost verbatim into an answer. So it should read like a directory entry, not a tagline. State what you sell, who it is for, and what makes you different, plus one or two details a reader can check.

Here is the actual intro paragraph of this site’s own file, at https://seoedgeai.com/llms.txt:

SEOEdgeAI is an AI SEO agent for Cloudflare sites that connects in two seconds and autonomously rewrites titles, meta descriptions, structured data and articles, publishes articles, and builds backlinks while learning from Google Search Console results. It operates via a Cloudflare Worker and edge proxy, requires no code or DNS change, and is offered in free and paid plans.

Every sentence carries a verifiable claim: what the product is, who it is for, how it is different, how it is delivered, and what it costs. There are no superlatives, no “leading” or “best-in-class,” and nothing an agent could quote that a customer could not check. That is the test to apply to your own intro: if you cannot verify a sentence on your site or your about page, cut it.

An llms.txt template you can copy

# Acme Analytics

> Acme Analytics is a usage analytics platform for B2B SaaS companies.
> It tracks feature adoption, retention and churn in one dashboard, and
> keeps all data on the customer's own cloud. Founded in 2021, it is
> used by about 400 SaaS teams, mostly in Europe and North America.

## What this site covers

- Usage analytics for SaaS
- Retention and churn benchmarks
- Product-led growth metrics

## Pages

- [Product](https://acme.example.com/product): Pricing and feature overview
- [Documentation](https://acme.example.com/docs): API reference and setup guides
- [Customer stories](https://acme.example.com/customers): How SaaS teams use Acme Analytics
- [About](https://acme.example.com/about): Company background and team

• The H1 is the only required element, so even a minimal file works.
• Each link is [name](url) with a note after the colon telling the agent what the page contains and when to fetch it. The note is what turns a list of links into a map.
• “What this site covers” is a topics list: it tells an agent which questions your pages can answer, which is what makes your site a candidate to be quoted on those questions.
• Keep the whole file under a screenful. If it grows, agents skim or skip it.

For a real-world file that demonstrates the subpath pattern, Cloudflare’s llms.txt is a good model: it links to one llms.txt per product area instead of trying to list everything in one place.

How to keep llms.txt updated

Treat it like your sitemap and check it whenever your site changes:

• Add pages as they become worth quoting; remove links that 404 before an agent hits them.
• Refresh the blockquote when your facts change (pricing, positioning, geography, headcount).
• Test it the way an agent does: fetch /llms.txt in a browser and read it cold, as if you had never seen the site.
• Automate it if you can. If your publishing pipeline generates the site, render llms.txt at build time.
• Since May 2026, Chrome’s Lighthouse has included an “Agentic Browsing” audit that tries to fetch /llms.txt and flags server errors, so a serving problem is now caught by a standard audit instead of silent neglect.

If your site runs on Cloudflare, SEOEdgeAI generates an llms.txt file that answer engines can read for every connected site as part of its on-autopilot SEO service, and the file quoted above is a live example of the result.

Why being quoted in AI answers is increasingly the point

This is the context that makes the file worth maintaining. Adobe Analytics measured referral traffic from generative AI sources to retail websites in the United States growing by more than 1,200% between July 2024 and February 2025, with a peak near 1,950% during Cyber Monday 2024, CMSWire reports. Buyers complete their research inside an assistant and arrive at a site with a question mostly answered; being the page the assistant recommends, and quotes, is the new version of ranking.

llms.txt does not do that work alone. It makes the work possible by giving the agent something clean and specific to read, and by telling it which of your pages answer which questions. Combined with content that states its facts plainly (real numbers, dates, named sources, no filler), it is the difference between being readable to a model and being quotable in its answer. Google ignores the file for ranking, and llms.txt will not rescue thin pages, but nobody quotes a page they cannot parse.

The wider shift, from search results to agent-driven answers, is the subject of our complete guide to AI SEO agents in 2026. The one-paragraph version: an AI SEO agent reads your site and its Search Console data, decides what to change, applies it at the edge, and measures whether it worked. An llms.txt file is one of the cleanest, cheapest outputs of that loop: a first-party file that makes your site legible to every agent and answer engine that bothers to check.