--- title: "llms.txt Insights · How 523+ Projects Structure Their AI Sitemaps · mdream" canonical_url: "https://mdream.dev/insights" last_updated: "2026-08-29T21:03:59.902Z" meta: description: "Data-driven analysis of 523+ real-world llms.txt files. See size distributions, section patterns, link coverage, compliance scores, and best practice adoption across the ecosystem." "og:description": "Analyze how 523+ projects structure their llms.txt. Size distributions, section patterns, compliance scores, and actionable recommendations." "og:title": "llms.txt Insights: How 523+ Projects Structure Their AI Sitemaps" --- [llms.txt Tools](https://mdream.dev/llms-txt) # **llms.txt Insights ** Data from 523 real-world llms.txt files, analyzed for structure, size, compliance, and patterns. Last updated March 30, 2026 Projects **523** Total Links **151,268** Median Size **15.6 KB** Unique Sections **5,256** ## File Size Distribution How large are llms.txt files across the ecosystem? The smallest is 213 B (/llms.txt directory), the largest 1.8 MB (What is Datadog?). < 1 KB **18** 1–5 KB **98** 5–20 KB **187** 20–50 KB **92** 50–100 KB **52** 100–500 KB **64** 500 KB+ **12** Min **213 B** Median **15.6 KB** Max **1.8 MB** ## Best Practices Adoption How well do real-world llms.txt files follow the spec and best practices? Under 100K tokens **97%** Has H2 sections **96%** No code blocks **92%** Has H1 title **90%** No H3+ headings **78%** Uses absolute URLs **77%** Under 50 KB **76%** Has blockquote description **50%** All links described **35%** Has Optional section **23%** ## Link Description Coverage Across 151,268 links, only 54% include descriptions. Descriptions give LLMs context about each page without fetching it. **54%**described With descriptions: **81,194** Without descriptions: **70,074** Median: 83 links per file ## Most Common Section Names 5,256 unique H2 section names across all files. These are the most popular. "Docs" 177x "Installation" 136x "API Reference" 130x "Optional" 121x "Usage" 94x "OpenAPI Specs" 87x "Getting started" 83x "Next steps" 76x "Example" 47x "Troubleshooting" 46x "Install" 45x "How it works" 44x ## URL Patterns URL Style 85% absolute15% relative File Placement /llms.txt (root)**87%** /docs/llms.txt**13%** Other paths**1%** Link Target Extensions .md101,691.none57,235.txt9,719.mdx6,614.html1,688.png222.com188.json187 ## By Category How do different categories of projects approach llms.txt? llms.txt usage statistics by project category | Category | Count | Avg Size | Avg Links | Size Distribution | | --- | --- | --- | --- | --- | | Libraries | 232 | 71.6 KB | 307 | | | AI | 54 | 67.9 KB | 170 | | | Documentation Tools | 51 | 36.3 KB | 208 | | | UI Libraries | 37 | 19.5 KB | 100 | | | Frameworks | 34 | 64.7 KB | 245 | | | Databases | 30 | 50.5 KB | 294 | | | Platforms | 28 | 143.5 KB | 354 | | | Developer Tools | 18 | 37.8 KB | 160 | | | Vue Ecosystem | 10 | 16.4 KB | 117 | | | CMS | 9 | 24.4 KB | 180 | | | Build Tools & Runtimes | 8 | 34.6 KB | 318 | | | Monitoring | 5 | 431.7 KB | 3161 | | | Authentication | 3 | 214.6 KB | 1470 | | | E-commerce | 3 | 54.0 KB | 300 | | | React Ecosystem | 1 | 14.0 KB | 177 | | ## Structural Patterns Has blockquote description**50%** References llms-full.txt**10%** Has "Optional" section**23%** Uses H3+ headings (non-spec)**22%** 19 projects use a flat structure (no H2 sections): Next.js, Ark UI, Supabase, Ri Xu Online, Appwrite, Kotlin Documentation, Strapi Documentation, http://docs.copilotkit.ai llms.txt, https://docs.litellm.ai/ llms.txt, Documentation, Memvid, https://www.sardine.ai llms.txt, Alpaca Main Page, sitespeak.ai llms.txt, https://tolgee.io llms.txt, https://latte.nette.org/ llms.txt, WordLift, https://nette.org/en/ llms.txt, ScreenshotOne.com Documentation 45 projects have multiple H1 headings: Svelte, SvelteKit, Ark UI, SWR, Remotion, Turso... 44 projects include code blocks: Svelte, SvelteKit, Waku, daisyUI, Zustand, SWR... ## Token Estimates How much LLM context do these files consume? Min **54** Median **3,992** Mean **17,090** Max **471,765** 18 files exceed 100K tokens What is Datadog?: 471,765 tokens Documentation: 447,604 tokens Add a New Provider: 371,002 tokens LLMs.txt - Sitemap for AI content discovery: 307,831 tokens Tamagui Complete Documentation: 267,532 tokens Kilo Code Documentation: 254,109 tokens Snowflake Documentation: 231,479 tokens Gemini CLI Documentation: 219,925 tokens PraisonAI: 209,780 tokens SvelteKit: 137,599 tokens Vultr Documentation: 135,514 tokens Mailchimp - Complete Web Content Analysis: 131,208 tokens Modular: 125,191 tokens Svelte: 117,430 tokens LaunchDarkly | Documentation: 115,318 tokens Agno: 114,069 tokens Documentation: 111,033 tokens Auth0 blog: 100,823 tokens ## Compliance Leaderboard Top Compliance [1.Cloudflare Workers**10/10**](https://mdream.dev/llms-txt/cloudflare-workers) [2.Trigger.dev**10/10**](https://mdream.dev/llms-txt/trigger-dev) [3.reveal.js Documentation**10/10**](https://mdream.dev/llms-txt/reveal-js-documentation) [4.Javalin**10/10**](https://mdream.dev/llms-txt/javalin) [5.Milvus**10/10**](https://mdream.dev/llms-txt/milvus) [6.Magic UI**10/10**](https://mdream.dev/llms-txt/magicui) [7.SurrealDB**10/10**](https://mdream.dev/llms-txt/surrealdb) [8.Super Productivity**10/10**](https://mdream.dev/llms-txt/super-productivity) Needs Improvement [Kilo Code Documentation**1/10**](https://mdream.dev/llms-txt/kilocode) [Gemini CLI Documentation**1/10**](https://mdream.dev/llms-txt/gemini-cli) [SvelteKit**1/10**](https://mdream.dev/llms-txt/sveltekit) [Svelte**1/10**](https://mdream.dev/llms-txt/svelte) [Preact Documentation**2/10**](https://mdream.dev/llms-txt/preactjs) [--banner:js hack from https://github.com/evanw/esbuild/pull/2067**2/10**](https://mdream.dev/llms-txt/middy) [Hurl Documentation**2/10**](https://mdream.dev/llms-txt/hurl) [Tamagui Complete Documentation**2/10**](https://mdream.dev/llms-txt/tamagui) [Add a New Provider**2/10**](https://mdream.dev/llms-txt/novu) ## Size Extremes Largest Files [What is Datadog?1.8 MB](https://mdream.dev/llms-txt/datadoghq) [Add a New Provider1.4 MB](https://mdream.dev/llms-txt/novu) [Tamagui Complete Documentation1.0 MB](https://mdream.dev/llms-txt/tamagui) [Kilo Code Documentation992.6 KB](https://mdream.dev/llms-txt/kilocode) [Snowflake Documentation904.2 KB](https://mdream.dev/llms-txt/snowflake) [Gemini CLI Documentation859.1 KB](https://mdream.dev/llms-txt/gemini-cli) [PraisonAI819.5 KB](https://mdream.dev/llms-txt/praisonai) [SvelteKit537.5 KB](https://mdream.dev/llms-txt/sveltekit) Smallest Files [/llms.txt directory213 B](https://mdream.dev/llms-txt/llms-txt-directory) [Video.js v10229 B](https://mdream.dev/llms-txt/video-js) [Three.js274 B](https://mdream.dev/llms-txt/three-js) [Sherlock Project523 B](https://mdream.dev/llms-txt/sherlock) [Supabase546 B](https://mdream.dev/llms-txt/supabase) [WireMock560 B](https://mdream.dev/llms-txt/wiremock) [LITIENGINE 🎮: The pure 2D Java Game Engine596 B](https://mdream.dev/llms-txt/litiengine-the-pure-2d-java-game-engine) [Terragrunt615 B](https://mdream.dev/llms-txt/terragrunt) [llms.txt648 B](https://mdream.dev/llms-txt/llmstxt-org) ## Recommendations from the Data **Describe every link ** 53% of links across the ecosystem lack descriptions. Adding a short description after each link (e.g. `- [Page](url): What this page covers`) dramatically improves LLM comprehension and saves token budget. **Target 5 to 50 KB ** The median file is 16 KB. Files under 50 KB fit easily in any LLM context. Larger files should consider an "Optional" section or splitting into llms.txt (index) and llms-full.txt (full content). Only 12% of files use the Optional pattern. **Use standard section names ** "Docs", "Guides", "Reference", "Examples", and "Getting Started" are the most recognized patterns. Using familiar names helps LLMs navigate your file structure predictably. Avoid deeply nested H3+ headings (27% of files violate this). **Always use absolute URLs ** 21% of projects use relative URLs. Relative URLs break when the file is consumed outside its original domain (AI agents, documentation aggregators, IDE integrations). Always use full https:// URLs. **Consider llms-full.txt ** 28% of projects reference an llms-full.txt companion file. This pattern is growing because it enables RAG pipelines and deep context loading without requiring the LLM to fetch each page individually. ## Frequently Asked Questions We track and analyze llms.txt files from 523+ projects across frameworks, libraries, platforms, databases, and developer tools. The standard has seen rapid adoption since its introduction, particularly among developer documentation sites. The median llms.txt file is around 16 KB with about 3,992 tokens. This fits comfortably in any LLM context window. Files over 100 KB start to consume significant context, and files over 500 KB may exceed smaller model limits. Aim for 5 to 50 KB with well-described links. The most common section names are "Docs", "Guides", "Reference", "Examples", and "Getting Started". Most effective files use 3 to 8 sections. An "Optional" section is a useful pattern that lets LLMs skip less critical content when context is limited. Yes. Only 54% of links across all analyzed files include descriptions, but descriptions dramatically improve AI comprehension. A link description tells the LLM what the page covers without needing to fetch it, saving tokens and improving accuracy. About 10% of analyzed projects reference an llms-full.txt companion file. This file embeds the full content of linked pages inline, so AI systems can consume everything in one request. It is especially valuable for RAG pipelines and deep context loading. We fetch and parse every llms.txt file from our registry of 523+ projects, validate structure against the spec, extract statistics on size, sections, links, and URL patterns, then compute compliance scores. The data is regenerated periodically to track changes. ## Related Tools [**llms.txt Generator**Generate an llms.txt for your own site.](https://mdream.dev/tools/llms-txt/generator) [**llms.txt Validator**Validate your file against the spec.](https://mdream.dev/tools/llms-txt/validator)