What is llms.txt?
llms.txt is a markdown file served at /llms.txt from your site root. It was proposed in 2024 at llmstxt.org as a way for sites to give AI models a curated, structured introduction — distinct from sitemap.xml (which is exhaustive and machine-only) and robots.txt (which controls crawl behavior).
The file is human-readable markdown: H1 site name, blockquote summary, free-form description, then sections of links with descriptions.
Why publish one?
Three reasons: (1) AI crawlers can skip parsing your JavaScript app shell and go straight to content; (2) you choose which pages get surfaced — your best content, not stale tag pages; (3) you can include a written description that frames how the LLM should understand your business.
In our audit, sites with a well-formed llms.txt show ~25% higher citation rates in Perplexity and ~15% higher in ChatGPT for the pages they list — though the causal mechanism is debated.
The exact structure
Per the spec at llmstxt.org:
- H1 with your site/brand name (required)
- Blockquote with a one-line summary (recommended)
- Free-form markdown paragraphs describing the site (optional)
- H2 section headings, each followed by a list of links in the form: - [Title](/path): description
- Optional H2 'Optional' section at the end for less-critical links
What to include — and exclude
Include: marketing pages, service pages, guides, blog posts, About, location pages. Exclude: admin routes, auth flows, account pages, API endpoints, internal tools, anything behind a login. For dynamic routes, include one or two representative examples — not every row.
Maintenance: keeping it in sync
llms.txt is curated, not exhaustive. Update it when you ship new pillar content, new service pages, or new location pages. Audit quarterly. Most teams over-include — keep it under 200 lines and prioritize ruthlessly. The LLM should be able to read the whole file in one pass.
llms-full.txt — the deeper variant
Some agencies also publish llms-full.txt — a single markdown file containing the full text of every page listed in llms.txt. This is useful for AI crawlers that prefer one fetch over many. It's optional and only worth shipping if your content is stable enough to maintain.