Weekly report · 2026-W20
The state of AI crawling
Measured on 29,998 reachable domains for the week ending 2026-05-17. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
29,998
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 9.8% | 2,937 of 29,991 | new |
| CCBot | Common Crawl Foundation | archive | 9.6% | 2,881 of 29,991 | new |
| Bytespider | ByteDance | training | 9.4% | 2,817 of 29,991 | new |
| ClaudeBot | Anthropic | training | 8.6% | 2,574 of 29,991 | new |
| Meta-ExternalAgent | Meta | training | 7.8% | 2,340 of 29,991 | new |
| Amazonbot | Amazon | training | 7.7% | 2,298 of 29,991 | new |
| Applebot-Extended | Apple | training | 7.6% | 2,291 of 29,991 | new |
| Google-Extended | training | 7.6% | 2,269 of 29,991 | new | |
| anthropic-ai | Anthropic | training | 7.3% | 2,193 of 29,991 | new |
| omgilibot | Webz.io | training | 7.2% | 2,164 of 29,991 | new |
| Diffbot | Diffbot | training | 7.1% | 2,120 of 29,991 | new |
| cohere-ai | Cohere | live retrieval | 7.0% | 2,112 of 29,991 | new |
| omgili | Webz.io | training | 6.9% | 2,055 of 29,991 | new |
| Google-CloudVertexBot | live retrieval | 4.5% | 1,361 of 29,991 | new | |
| GoogleOther | training | 4.4% | 1,314 of 29,991 | new | |
| MistralAI-Training | Mistral AI | training | 3.8% | 1,130 of 29,991 | new |
| SemrushBot-OCOB | Semrush | training | 3.5% | 1,059 of 29,991 | new |
| PerplexityBot | Perplexity | answer index | 3.4% | 1,006 of 29,991 | new |
| ChatGPT-User | OpenAI | live retrieval | 3.3% | 983 of 29,991 | new |
| FacebookBot | Meta | training | 3.1% | 919 of 29,991 | new |
| YouBot | You.com | answer index | 2.9% | 876 of 29,991 | new |
| ImagesiftBot | ImageSift (Hive) | training | 2.7% | 817 of 29,991 | new |
| Timpibot | Timpi | training | 2.7% | 811 of 29,991 | new |
| Scrapy | Zyte (open-source framework) | training | 2.4% | 706 of 29,991 | new |
| AI2Bot | Allen Institute for AI | training | 2.3% | 692 of 29,991 | new |
| cohere-training-data-crawler | Cohere | training | 2.1% | 615 of 29,991 | new |
| OAI-SearchBot | OpenAI | answer index | 2.0% | 608 of 29,991 | new |
| Meta-ExternalFetcher | Meta | live retrieval | 1.8% | 553 of 29,991 | new |
| PanguBot | Huawei | training | 1.7% | 502 of 29,991 | new |
| Claude-User | Anthropic | live retrieval | 1.7% | 498 of 29,991 | new |
| DuckAssistBot | DuckDuckGo | live retrieval | 1.7% | 498 of 29,991 | new |
| Perplexity-User | Perplexity | live retrieval | 1.6% | 476 of 29,991 | new |
| Claude-SearchBot | Anthropic | answer index | 1.5% | 463 of 29,991 | new |
| MistralAI-User | Mistral AI | live retrieval | 1.5% | 457 of 29,991 | new |
| Kangaroo Bot | Kangaroo LLM | training | 1.4% | 433 of 29,991 | new |
| Meta-WebIndexer | Meta | answer index | 1.3% | 402 of 29,991 | new |
| Applebot | Apple | answer index | 1.2% | 350 of 29,991 | new |
| Amzn-SearchBot | Amazon | answer index | 0.8% | 253 of 29,991 | new |
| TikTokSpider | ByteDance | live retrieval | 0.8% | 247 of 29,991 | new |
| Amzn-User | Amazon | live retrieval | 0.7% | 208 of 29,991 | new |
| MistralAI-Index | Mistral AI | answer index | 0.4% | 111 of 29,991 | new |
| facebookexternalhit | Meta | live retrieval | 0.2% | 56 of 29,991 | new |
| Meta-ExternalAds | Meta | training | 0.1% | 41 of 29,991 | new |
| Googlebot-News | answer index | 0.1% | 39 of 29,991 | new | |
| Bingbot | Microsoft | answer index | 0.1% | 35 of 29,991 | new |
| msnbot | Microsoft | answer index | 0.1% | 18 of 29,991 | new |
| Googlebot | answer index | 0.1% | 17 of 29,991 | new | |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 13 of 29,991 | new |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 0 total
No crawler policy changes were recorded in this week.
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2026-W20", 2026-05-17. https://crawlcensus.com/report/2026-W20
Machine-readable: census totals, full corpus, crawler registry. Method: methodology.