llms.txt Insights
Data from 523 real-world llms.txt files, analyzed for structure, size, compliance, and patterns. Last updated March 30, 2026
File Size Distribution
How large are llms.txt files across the ecosystem? The smallest is 213 B (/llms.txt directory), the largest 1.8 MB (What is Datadog?).
Best Practices Adoption
How well do real-world llms.txt files follow the spec and best practices?
Link Description Coverage
Across 151,268 links, only 54% include descriptions. Descriptions give LLMs context about each page without fetching it.
Most Common Section Names
5,256 unique H2 section names across all files. These are the most popular.
URL Patterns
By Category
How do different categories of projects approach llms.txt?
| Category | Count | Avg Size | Avg Links | Size Distribution |
|---|---|---|---|---|
| Libraries | 232 | 71.6 KB | 307 | |
| AI | 54 | 67.9 KB | 170 | |
| Documentation Tools | 51 | 36.3 KB | 208 | |
| UI Libraries | 37 | 19.5 KB | 100 | |
| Frameworks | 34 | 64.7 KB | 245 | |
| Databases | 30 | 50.5 KB | 294 | |
| Platforms | 28 | 143.5 KB | 354 | |
| Developer Tools | 18 | 37.8 KB | 160 | |
| Vue Ecosystem | 10 | 16.4 KB | 117 | |
| CMS | 9 | 24.4 KB | 180 | |
| Build Tools & Runtimes | 8 | 34.6 KB | 318 | |
| Monitoring | 5 | 431.7 KB | 3161 | |
| Authentication | 3 | 214.6 KB | 1470 | |
| E-commerce | 3 | 54.0 KB | 300 | |
| React Ecosystem | 1 | 14.0 KB | 177 |
Structural Patterns
Token Estimates
How much LLM context do these files consume?
Compliance Leaderboard
Size Extremes
Recommendations from the Data
53% of links across the ecosystem lack descriptions. Adding a short description after each link (e.g. - [Page](url): What this page covers) dramatically improves LLM comprehension and saves token budget.
The median file is 16 KB. Files under 50 KB fit easily in any LLM context. Larger files should consider an "Optional" section or splitting into llms.txt (index) and llms-full.txt (full content). Only 12% of files use the Optional pattern.
"Docs", "Guides", "Reference", "Examples", and "Getting Started" are the most recognized patterns. Using familiar names helps LLMs navigate your file structure predictably. Avoid deeply nested H3+ headings (27% of files violate this).
21% of projects use relative URLs. Relative URLs break when the file is consumed outside its original domain (AI agents, documentation aggregators, IDE integrations). Always use full https:// URLs.
28% of projects reference an llms-full.txt companion file. This pattern is growing because it enables RAG pipelines and deep context loading without requiring the LLM to fetch each page individually.
Frequently Asked Questions
We track and analyze llms.txt files from 523+ projects across frameworks, libraries, platforms, databases, and developer tools. The standard has seen rapid adoption since its introduction, particularly among developer documentation sites.
The median llms.txt file is around 16 KB with about 3,992 tokens. This fits comfortably in any LLM context window. Files over 100 KB start to consume significant context, and files over 500 KB may exceed smaller model limits. Aim for 5 to 50 KB with well-described links.
The most common section names are "Docs", "Guides", "Reference", "Examples", and "Getting Started". Most effective files use 3 to 8 sections. An "Optional" section is a useful pattern that lets LLMs skip less critical content when context is limited.
Yes. Only 54% of links across all analyzed files include descriptions, but descriptions dramatically improve AI comprehension. A link description tells the LLM what the page covers without needing to fetch it, saving tokens and improving accuracy.
About 10% of analyzed projects reference an llms-full.txt companion file. This file embeds the full content of linked pages inline, so AI systems can consume everything in one request. It is especially valuable for RAG pipelines and deep context loading.
We fetch and parse every llms.txt file from our registry of 523+ projects, validate structure against the spec, extract statistics on size, sections, links, and URL patterns, then compute compliance scores. The data is regenerated periodically to track changes.