Guide

AI crawlers in 2026

Nine AI crawlers account for virtually all agent-originated traffic in 2026. Blanket-blocking them means no citations from ChatGPT, Claude, Perplexity, or Google AI Overviews. Block selectively with robots.txt and guide the rest with llms.txt.

Updated 2026-04-22

Check this on your site

AI search visibility audit

One-click audit: llms.txt, Accept-header, robots.txt, sitemap.md, token savings.

Related guides

Generative engine optimization (GEO)

Generative engine optimization (GEO) is the discipline of getting your content cited by ChatGPT, Claude, Perplexity, and Google AI Overviews. It blends classic SEO, structured data, and new formats like llms.txt + Accept-header markdown.

Answer engine optimization (AEO)

Answer engine optimization (AEO) is the practice of structuring content so that ChatGPT, Claude, Perplexity, Google AI Overviews, and voice assistants pick it as the canonical answer.

Markdown for agents

AI agents read markdown 2–5× more efficiently than HTML. Serve it via Accept-header negotiation, sitemap.md, and <link rel="alternate" type="text/markdown"> and every agent-originated request instantly becomes cheaper and smarter.