Weekly report · 2026-W45
The state of AI crawling
Measured on 29,997 reachable domains for the week ending 2026-11-08. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 9.8% | 2,931 of 29,967 | +0.1 pt |
| CCBot | Common Crawl Foundation | archive | 9.6% | 2,870 of 29,967 | +0.1 pt |
| Bytespider | ByteDance | training | 9.4% | 2,804 of 29,967 | +0.2 pt |
| ClaudeBot | Anthropic | training | 8.6% | 2,568 of 29,967 | +0.2 pt |
| Meta-ExternalAgent | Meta | training | 7.8% | 2,334 of 29,967 | +0.2 pt |
| Amazonbot | Amazon | training | 7.6% | 2,283 of 29,967 | +0.1 pt |
| Applebot-Extended | Apple | training | 7.6% | 2,280 of 29,967 | +0.2 pt |
| Google-Extended | training | 7.6% | 2,263 of 29,967 | +0.1 pt | |
| anthropic-ai | Anthropic | training | 7.3% | 2,183 of 29,967 | +0.2 pt |
| omgilibot | Webz.io | training | 7.2% | 2,160 of 29,967 | +0.2 pt |
| Diffbot | Diffbot | training | 7.1% | 2,115 of 29,967 | +0.2 pt |
| cohere-ai | Cohere | live retrieval | 7.0% | 2,109 of 29,967 | +0.2 pt |
| omgili | Webz.io | training | 6.8% | 2,051 of 29,967 | +0.2 pt |
| Google-CloudVertexBot | live retrieval | 4.5% | 1,356 of 29,967 | +0.2 pt | |
| GoogleOther | training | 4.4% | 1,308 of 29,967 | +0.2 pt | |
| MistralAI-Training | Mistral AI | training | 3.7% | 1,117 of 29,967 | +0.2 pt |
| SemrushBot-OCOB | Semrush | training | 3.5% | 1,062 of 29,967 | level |
| PerplexityBot | Perplexity | answer index | 3.4% | 1,013 of 29,967 | level |
| ChatGPT-User | OpenAI | live retrieval | 3.3% | 990 of 29,967 | level |
| FacebookBot | Meta | training | 3.1% | 923 of 29,967 | level |
| YouBot | You.com | answer index | 2.9% | 883 of 29,967 | level |
| ImagesiftBot | ImageSift (Hive) | training | 2.7% | 821 of 29,967 | level |
| Timpibot | Timpi | training | 2.7% | 815 of 29,967 | level |
| Scrapy | Zyte (open-source framework) | training | 2.4% | 713 of 29,967 | level |
| AI2Bot | Allen Institute for AI | training | 2.3% | 698 of 29,967 | level |
| cohere-training-data-crawler | Cohere | training | 2.1% | 620 of 29,967 | level |
| OAI-SearchBot | OpenAI | answer index | 2.0% | 614 of 29,967 | level |
| Meta-ExternalFetcher | Meta | live retrieval | 1.9% | 560 of 29,967 | level |
| PanguBot | Huawei | training | 1.7% | 506 of 29,967 | level |
| Claude-User | Anthropic | live retrieval | 1.7% | 503 of 29,967 | level |
| DuckAssistBot | DuckDuckGo | live retrieval | 1.7% | 503 of 29,967 | level |
| Perplexity-User | Perplexity | live retrieval | 1.6% | 483 of 29,967 | level |
| Claude-SearchBot | Anthropic | answer index | 1.6% | 469 of 29,967 | level |
| MistralAI-User | Mistral AI | live retrieval | 1.5% | 463 of 29,967 | level |
| Kangaroo Bot | Kangaroo LLM | training | 1.5% | 439 of 29,967 | level |
| Meta-WebIndexer | Meta | answer index | 1.3% | 404 of 29,967 | level |
| Applebot | Apple | answer index | 1.2% | 352 of 29,967 | level |
| Amzn-SearchBot | Amazon | answer index | 0.9% | 255 of 29,967 | level |
| TikTokSpider | ByteDance | live retrieval | 0.8% | 247 of 29,967 | level |
| Amzn-User | Amazon | live retrieval | 0.7% | 208 of 29,967 | level |
| MistralAI-Index | Mistral AI | answer index | 0.4% | 112 of 29,967 | level |
| facebookexternalhit | Meta | live retrieval | 0.2% | 57 of 29,967 | level |
| Googlebot-News | answer index | 0.1% | 39 of 29,967 | level | |
| Meta-ExternalAds | Meta | training | 0.1% | 39 of 29,967 | level |
| Bingbot | Microsoft | answer index | 0.1% | 35 of 29,967 | level |
| msnbot | Microsoft | answer index | 0.1% | 19 of 29,967 | level |
| Googlebot | answer index | 0.1% | 17 of 29,967 | level | |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 14 of 29,967 | level |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 200 total
- shopee.ph ClaudeBot is no longer refused at the network edge
- themoviedb.org Applebot is now allowed by robots.txt
- themoviedb.org facebookexternalhit is now allowed by robots.txt
- harknmav.fun AI access score moved 65 to 46
- wps.cn AI access score moved 56 to 66
- moengage.com GPTBot is now refused at the network edge, whatever robots.txt permits
- moengage.com ClaudeBot is now refused at the network edge, whatever robots.txt permits
- oreilly.com GPTBot is now allowed by robots.txt
- oreilly.com anthropic-ai is now allowed by robots.txt
- oreilly.com Meta-ExternalAgent is now allowed by robots.txt
- oreilly.com Bytespider is now allowed by robots.txt
- oreilly.com Amazonbot is now allowed by robots.txt
- oreilly.com CCBot is now allowed by robots.txt
- oreilly.com Diffbot is now allowed by robots.txt
- oreilly.com omgili is now allowed by robots.txt
- oreilly.com omgilibot is now allowed by robots.txt
- oreilly.com AI2Bot is now allowed by robots.txt
- oreilly.com cohere-ai is now allowed by robots.txt
- oreilly.com Timpibot is now allowed by robots.txt
- oreilly.com Removed its llms.txt file
- oreilly.com AI access score moved 94 to 54
- saveetha.in AI access score moved 69 to 83
- uchicago.edu GPTBot is no longer refused at the network edge
- uchicago.edu OAI-SearchBot is no longer refused at the network edge
- uchicago.edu ClaudeBot is no longer refused at the network edge
- uchicago.edu PerplexityBot is no longer refused at the network edge
- uchicago.edu AI access score moved 66 to 83
- scientificamerican.com OAI-SearchBot is no longer refused at the network edge
- scientificamerican.com ClaudeBot is no longer refused at the network edge
- scientificamerican.com PerplexityBot is no longer refused at the network edge
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2026-W45", 2026-11-08. https://crawlcensus.com/report/2026-W45
Machine-readable: census totals, full corpus, crawler registry. Method: methodology.