What is llms.txt?
The complete guide to the llms.txt standard. Learn what it is, why your site needs one, and how to create it.
What is llms.txt?
llms.txt is a proposed standard that gives AI systems a structured overview of your website. Placed at the root of your domain (e.g. example.com/llms.txt), it acts as an AI sitemap: a curated, markdown-formatted index of your most important pages with descriptions and section headings.
Created by Jeremy Howard and published at llmstxt.org, the standard has been adopted by hundreds of companies. Our analysis of 95+ real-world files shows the median file is 16 KB with 94 links across 3 sections.
Why does llms.txt matter?
AI systems like ChatGPT, Claude, Perplexity, and Gemini increasingly browse the web to answer questions. Without llms.txt, these systems must crawl your entire site to understand what you offer. With llms.txt, they get a structured overview instantly.
For developer tools and documentation sites, llms.txt is especially valuable: AI coding assistants like Cursor, Windsurf, and Claude Code use it to write more accurate code by understanding your API surface and documentation structure.
How is llms.txt different from robots.txt?
robots.txt controls crawler access by telling bots which pages to exclude. llms.txt is the complement: it tells AI systems which pages to include and how they relate. robots.txt is about exclusion; llms.txt is about structured inclusion. Use both together.
Similarly, sitemap.xml lists URLs for search engines to index. llms.txt adds context: what each page covers, how pages relate, and which ones matter most. Think of it as a curated, AI-readable table of contents.
How is llms.txt different from sitemap.xml?
sitemap.xml is a flat list of URLs for search engine crawlers. It says "these pages exist." llms.txt is a structured, curated index for AI systems. It says "these pages matter, here is how they are organized, and here is what each one covers."
A large site might have 10,000 URLs in its sitemap but only 30 to 50 carefully curated links in its llms.txt. Across 95+ analyzed files, the median is 94 links; top projects like Astro and Deno score 10/10 on compliance with just 11 links each.
llms.txt Format Rules
The llms.txt specification defines a simple markdown format. Follow these rules to create a valid file.
- 1
Single H1 heading
The file must start with exactly one # heading containing your project or company name. 86% of analyzed files get this right.
- 2
Blockquote summary
A > blockquote immediately after the H1 with a one to two sentence description. Only 53% of files include one.
- 3
H2 section headings
Organize links under ## headings like Docs, API, Guides, Tutorials, Examples. The most common names are "Docs", "Guides", and "Examples" across 95+ files.
- 4
Markdown link format
List resources as - [Title](url): Description. Only 47% of links in the wild include descriptions, but they dramatically improve LLM comprehension.
- 5
Absolute URLs
All links must use full URLs (https://...) so the file works anywhere. 21% of projects still use relative URLs, which break in AI tools.
- 6
UTF-8 plain text
Serve as text/plain or text/markdown. Must be valid Markdown, UTF-8 encoded.
llms.txt Example
Here is what a well-structured llms.txt looks like in practice.
# Acme Docs
Acme is an API platform for building payment workflows.
## Getting Started
- - Quickstart: Set up your first integration in 5 minutes
- - Authentication: API keys, OAuth, and webhook signing
## API Reference
- - Payments: Create, capture, and refund payments
- - Customers: Manage customer profiles and payment methods
- - Webhooks: Event notifications and retry logic
## Guides
## Optional
- - Changelog: Recent API changes and deprecations
Use Cases
AI Search Visibility
Perplexity, ChatGPT Browse, and Google AI Overviews use structured data to cite your content. llms.txt helps these systems find, understand, and reference your pages accurately.
AI Coding Assistants
Developers using Cursor, Windsurf, or Claude Code get better results when your docs have a clear llms.txt. The AI can navigate your API surface and write accurate code.
RAG Pipelines
Teams building retrieval-augmented generation systems use llms.txt as a structured index and llms-full.txt as a retrieval source for vector databases.
Documentation Discovery
Open source projects with llms.txt make it trivial for contributors and users to find the right documentation page through AI tools.
How to Add llms.txt to Your Site
- 1
Create a file named llms.txt in your public/ or root directory
- 2
Write the content following the format spec: H1, blockquote, H2 sections with markdown links
- 3
Deploy. The file should be accessible at https://yourdomain.com/llms.txt
Works with any framework or hosting provider. No dependencies required.
Frequently Asked Questions
llms.txt is a proposed standard for providing AI systems with a structured, markdown-formatted index of your website. Published at llmstxt.org by Jeremy Howard, it defines a format for listing your most important pages under section headings with descriptions, served at /llms.txt on your domain.
llms.txt is a concise index with links and descriptions. llms-full.txt is an optional companion that embeds the full content of each linked page inline, so AI systems can consume everything in one request. llms.txt is for navigation; llms-full.txt is for deep context and RAG.
Not yet. llms.txt is a community proposal, not a W3C or IETF standard. However, it has seen rapid adoption across the industry. Companies like Anthropic, Stripe, Cloudflare, and Vercel all publish llms.txt files.
You can write one manually following the format (H1, blockquote, H2 sections with markdown links) or use our free llms.txt generator to crawl your site and create a starting point. For Nuxt sites, the @mdream/nuxt module generates it automatically.
Place it at the root of your domain so it is accessible at https://yourdomain.com/llms.txt. Most web frameworks support serving static files from a public directory. For dynamic generation, create a route handler that outputs the file.
The median across 95+ analyzed files is 94 links, but the most compliant files tend to be more focused. Aim for 20 to 100 focused links with descriptions. AI systems have context limits (the median file is ~4,000 tokens), so a curated file outperforms a massive list.
Google has not officially endorsed llms.txt as a ranking signal. However, Google AI Overviews and other AI-powered search features can benefit from the structured content that llms.txt provides. The primary value is for AI systems, not traditional SEO.
Yes. Use our free llms.txt validator to check your file against the official specification. It validates heading hierarchy, link formatting, blockquote summary, and other structural requirements.