AI crawlers · Who crawls your site and how to control them · mdream

[All guides](https://mdream.dev/guides/ai-crawlers/guides)

Guide

# AI crawlers in 2026

Nine AI crawlers account for virtually all agent-originated traffic in 2026. Blanket-blocking them means no citations from ChatGPT, Claude, Perplexity, or Google AI Overviews. Block selectively with robots.txt and guide the rest with llms.txt.

Updated 2026-04-22

## [The crawlers that matter](#the-crawlers-that-matter)

| Crawler | User-agent | Purpose | Who it feeds |
| --- | --- | --- | --- |
| GPTBot | `GPTBot` | training | OpenAI foundation models |
| ChatGPT-User | `ChatGPT-User` | on-demand browsing | ChatGPT |
| OAI-SearchBot | `OAI-SearchBot` | search index | SearchGPT |
| ClaudeBot | `ClaudeBot` | training + retrieval | Anthropic |
| Claude-User | `Claude-User` | Claude Code, live agent fetches | Claude products |
| anthropic-ai | `anthropic-ai` | legacy | Anthropic |
| PerplexityBot | `PerplexityBot` | retrieval | Perplexity |
| Perplexity-User | `Perplexity-User` | live agent fetches | Perplexity |
| Google-Extended | `Google-Extended` | training opt-out | Gemini, Vertex |
| CCBot | `CCBot` | Common Crawl | everyone indirectly |
| Bytespider | `Bytespider` | training | ByteDance/Doubao |
| Applebot-Extended | `Applebot-Extended` | training opt-out | Apple Intelligence |

## [Block vs allow](#block-vs-allow)

Default guidance in 2026:

- **Allow** user-facing agents: ChatGPT-User, Claude-User, Perplexity-User. Blocking them means your content can't be cited live.
- **Allow** search-style bots: OAI-SearchBot, PerplexityBot, Google-Extended.
- **Consider blocking** training bots: GPTBot, ClaudeBot, CCBot, Bytespider. Your content still shows up in live fetches; you just opt out of future training corpora.

This is a legal and commercial call; pick once, apply sitewide, revisit quarterly.

## [robots.txt template](#robotstxt-template)

```
# AI — training bots (opt-out unless you have a commercial reason)
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: anthropic-ai
Disallow: /

# AI — live agent fetchers (allow, so your content gets cited)
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# Default
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap.md
```

## [Check which bots actually fetched you](#check-which-bots-actually-fetched-you)

Server logs are authoritative. Grep for:

```
grep -E 'GPTBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User|OAI-SearchBot|Bytespider|CCBot|Google-Extended|Applebot-Extended' access.log
```

## [Pair with llms.txt](#pair-with-llmstxt)

robots.txt controls access; llms.txt gives the agents that are allowed in a structured map. [Generate yours in seconds](https://mdream.dev/guides/ai-crawlers/tools/llms-txt/generator).

## [Related](#related)

- [Markdown for agents](https://mdream.dev/guides/ai-crawlers/guides/markdown-for-agents)
- [AI search visibility audit](https://mdream.dev/guides/ai-crawlers/tools/ai-audit)

## Check this on your site

[<h3>AI search visibility audit</h3>One-click audit: llms.txt, Accept-header, robots.txt, sitemap.md, token savings.](https://mdream.dev/guides/ai-crawlers/tools/ai-audit)

## Related guides

[<h3>Generative engine optimization (GEO)</h3>Generative engine optimization (GEO) is the discipline of getting your content cited by ChatGPT, Claude, Perplexity, and Google AI Overviews. It blends classic SEO, structured data, and new formats like llms.txt + Accept-header markdown.](https://mdream.dev/guides/ai-crawlers/guides/generative-engine-optimization) [<h3>Answer engine optimization (AEO)</h3>Answer engine optimization (AEO) is the practice of structuring content so that ChatGPT, Claude, Perplexity, Google AI Overviews, and voice assistants pick it as the canonical answer.](https://mdream.dev/guides/ai-crawlers/guides/answer-engine-optimization) [<h3>Markdown for agents</h3>AI agents read markdown 2–5× more efficiently than HTML. Serve it via Accept-header negotiation, sitemap.md, and <link rel="alternate" type="text/markdown"> and every agent-originated request instantly becomes cheaper and smarter.](https://mdream.dev/guides/ai-crawlers/guides/markdown-for-agents)

© 2026 [Harlan Wilton](https://github.com/harlan-zw) · [MIT](https://github.com/harlan-zw/mdream/blob/main/license)

[GitHub](https://github.com/harlan-zw/mdream) [Discord](https://discord.com/invite/275MBUBvgP)