Markdown for agents
AI agents read markdown 2–5× more efficiently than HTML. Serve it via Accept-header negotiation, sitemap.md, and <link rel="alternate" type="text/markdown"> and every agent-originated request instantly becomes cheaper and smarter.
The case for markdown
An agent fetching a modern web page gets 80–200 KB of HTML back. 90% of that is nav, footer, cookie banners, React serialized state, and inline styles. Even after Readability-style extraction, the overhead is real: every angle bracket is a token, and tokens are money.
Serving markdown cuts that to the content, nothing else. Typical savings on a documentation page: 70% fewer tokens, 80% less bandwidth.
Three ways to serve markdown
1. Accept-header content negotiation
The spec-correct approach. On Accept: text/markdown, respond with Content-Type: text/markdown; charset=utf-8 and a Vary: Accept header.
GET /docs/getting-started HTTP/1.1
Accept: text/markdown, text/html;q=0.9
HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept
2. <link rel="alternate">
For agents that don't advertise preferences (like OpenAI Codex CLI), publish the markdown URL in your HTML head:
<link rel="alternate" type="text/markdown" href="/docs/getting-started.md">
Agents that parse HTML will follow the link and fetch the markdown version.
3. sitemap.md
A plain-markdown list of canonical URLs, analogous to sitemap.xml:
# Example Site
- [Home](/index.md)
- [Docs](/docs/index.md)
- [Getting Started](/docs/getting-started.md)
Place it at /sitemap.md. Agents crawling your site for RAG will prefer it over sitemap.xml.
Which agents respect markdown
| Agent | Sends Accept: text/markdown? | Follows rel="alternate"? |
|---|---|---|
| Claude Code | yes | no |
| Cursor | yes | no |
| Codex CLI | no | yes |
| ChatGPT | no | partial |
| Perplexity | no | no |
| Gemini | no | no |
Expect this list to expand as markdown adoption becomes table stakes.
Implementation checklist
- Generate an
to give agents a structured map. - Add
<link rel="alternate" type="text/markdown">on every HTML page that has a markdown twin. - Implement Accept negotiation at the edge. Cloudflare Workers and Vercel Edge Functions both make this trivial.
- Publish a sitemap.md.
to confirm all four are in place.