Weekly report · 2026-W34 · current
The state of AI crawling
Measured on 13,154 reachable domains for the week ending 2026-08-23. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
← 2026-W33 · this is the latest
13,154
domains measured
8%
block an answer engine
19%
refuse AI agents at the edge
13%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| CCBot | Common Crawl Foundation | archive | 12.0% | 1,460 of 12,129 | new |
| GPTBot | OpenAI | training | 11.8% | 1,431 of 12,129 | new |
| Bytespider | ByteDance | training | 11.5% | 1,392 of 12,129 | new |
| ClaudeBot | Anthropic | training | 10.5% | 1,275 of 12,129 | new |
| Meta-ExternalAgent | Meta | training | 9.5% | 1,148 of 12,129 | new |
| Google-Extended | training | 9.4% | 1,146 of 12,129 | new | |
| Applebot-Extended | Apple | training | 9.1% | 1,099 of 12,129 | new |
| Amazonbot | Amazon | training | 9.0% | 1,090 of 12,129 | new |
| omgilibot | Webz.io | training | 5.4% | 653 of 12,129 | new |
| anthropic-ai | Anthropic | training | 5.1% | 616 of 12,129 | new |
| Diffbot | Diffbot | training | 4.9% | 600 of 12,129 | new |
| cohere-ai | Cohere | live retrieval | 4.9% | 599 of 12,129 | new |
| omgili | Webz.io | training | 4.9% | 599 of 12,129 | new |
| PerplexityBot | Perplexity | answer index | 4.6% | 553 of 12,129 | new |
| ChatGPT-User | OpenAI | live retrieval | 4.4% | 529 of 12,129 | new |
| FacebookBot | Meta | training | 4.3% | 518 of 12,129 | new |
| YouBot | You.com | answer index | 3.6% | 442 of 12,129 | new |
| SemrushBot-OCOB | Semrush | training | 3.6% | 436 of 12,129 | new |
| Timpibot | Timpi | training | 3.6% | 434 of 12,129 | new |
| ImagesiftBot | ImageSift (Hive) | training | 3.4% | 416 of 12,129 | new |
| Scrapy | Zyte (open-source framework) | training | 2.9% | 352 of 12,129 | new |
| AI2Bot | Allen Institute for AI | training | 2.8% | 341 of 12,129 | new |
| OAI-SearchBot | OpenAI | answer index | 2.7% | 327 of 12,129 | new |
| cohere-training-data-crawler | Cohere | training | 2.6% | 318 of 12,129 | new |
| DuckAssistBot | DuckDuckGo | live retrieval | 2.4% | 296 of 12,129 | new |
| Meta-ExternalFetcher | Meta | live retrieval | 2.4% | 296 of 12,129 | new |
| Claude-User | Anthropic | live retrieval | 2.2% | 265 of 12,129 | new |
| Claude-SearchBot | Anthropic | answer index | 2.2% | 263 of 12,129 | new |
| Perplexity-User | Perplexity | live retrieval | 2.2% | 262 of 12,129 | new |
| PanguBot | Huawei | training | 2.1% | 255 of 12,129 | new |
| Kangaroo Bot | Kangaroo LLM | training | 2.0% | 243 of 12,129 | new |
| MistralAI-User | Mistral AI | live retrieval | 1.9% | 231 of 12,129 | new |
| Google-CloudVertexBot | live retrieval | 1.7% | 203 of 12,129 | new | |
| Meta-WebIndexer | Meta | answer index | 1.4% | 168 of 12,129 | new |
| Applebot | Apple | answer index | 1.3% | 156 of 12,129 | new |
| GoogleOther | training | 1.2% | 151 of 12,129 | new | |
| TikTokSpider | ByteDance | live retrieval | 0.9% | 106 of 12,129 | new |
| Amzn-SearchBot | Amazon | answer index | 0.8% | 103 of 12,129 | new |
| Amzn-User | Amazon | live retrieval | 0.6% | 70 of 12,129 | new |
| MistralAI-Index | Mistral AI | answer index | 0.4% | 45 of 12,129 | new |
| facebookexternalhit | Meta | live retrieval | 0.2% | 24 of 12,129 | new |
| MistralAI-Training | Mistral AI | training | 0.2% | 23 of 12,129 | new |
| Googlebot-News | answer index | 0.1% | 17 of 12,129 | new | |
| Bingbot | Microsoft | answer index | 0.1% | 12 of 12,129 | new |
| Googlebot | answer index | 0.1% | 8 of 12,129 | new | |
| msnbot | Microsoft | answer index | 0.1% | 8 of 12,129 | new |
| Meta-ExternalAds | Meta | training | 0.0% | 3 of 12,129 | new |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 1 of 12,129 | new |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 3 total
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2026-W34", 2026-08-23. https://crawlcensus.com/report/2026-W34Machine-readable: census totals, full corpus, crawler registry. Method: methodology.