Weekly report · 2026-W46
The state of AI crawling
Measured on 29,997 reachable domains for the week ending 2026-11-15. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 9.8% | 2,937 of 29,982 | level |
| CCBot | Common Crawl Foundation | archive | 9.6% | 2,878 of 29,982 | level |
| Bytespider | ByteDance | training | 9.4% | 2,812 of 29,982 | level |
| ClaudeBot | Anthropic | training | 8.6% | 2,574 of 29,982 | level |
| Meta-ExternalAgent | Meta | training | 7.8% | 2,340 of 29,982 | level |
| Amazonbot | Amazon | training | 7.6% | 2,290 of 29,982 | level |
| Applebot-Extended | Apple | training | 7.6% | 2,288 of 29,982 | level |
| Google-Extended | training | 7.6% | 2,268 of 29,982 | level | |
| anthropic-ai | Anthropic | training | 7.3% | 2,190 of 29,982 | level |
| omgilibot | Webz.io | training | 7.2% | 2,166 of 29,982 | level |
| Diffbot | Diffbot | training | 7.1% | 2,121 of 29,982 | level |
| cohere-ai | Cohere | live retrieval | 7.1% | 2,114 of 29,982 | level |
| omgili | Webz.io | training | 6.9% | 2,057 of 29,982 | level |
| Google-CloudVertexBot | live retrieval | 4.5% | 1,360 of 29,982 | level | |
| GoogleOther | training | 4.4% | 1,311 of 29,982 | level | |
| MistralAI-Training | Mistral AI | training | 3.7% | 1,124 of 29,982 | level |
| SemrushBot-OCOB | Semrush | training | 3.5% | 1,058 of 29,982 | level |
| PerplexityBot | Perplexity | answer index | 3.4% | 1,014 of 29,982 | level |
| ChatGPT-User | OpenAI | live retrieval | 3.3% | 991 of 29,982 | level |
| FacebookBot | Meta | training | 3.1% | 923 of 29,982 | level |
| YouBot | You.com | answer index | 2.9% | 882 of 29,982 | level |
| ImagesiftBot | ImageSift (Hive) | training | 2.7% | 818 of 29,982 | level |
| Timpibot | Timpi | training | 2.7% | 814 of 29,982 | level |
| Scrapy | Zyte (open-source framework) | training | 2.4% | 711 of 29,982 | level |
| AI2Bot | Allen Institute for AI | training | 2.3% | 697 of 29,982 | level |
| cohere-training-data-crawler | Cohere | training | 2.1% | 619 of 29,982 | level |
| OAI-SearchBot | OpenAI | answer index | 2.1% | 615 of 29,982 | level |
| Meta-ExternalFetcher | Meta | live retrieval | 1.9% | 560 of 29,982 | level |
| PanguBot | Huawei | training | 1.7% | 506 of 29,982 | level |
| Claude-User | Anthropic | live retrieval | 1.7% | 505 of 29,982 | level |
| DuckAssistBot | DuckDuckGo | live retrieval | 1.7% | 504 of 29,982 | level |
| Perplexity-User | Perplexity | live retrieval | 1.6% | 483 of 29,982 | level |
| Claude-SearchBot | Anthropic | answer index | 1.6% | 469 of 29,982 | level |
| MistralAI-User | Mistral AI | live retrieval | 1.5% | 464 of 29,982 | level |
| Kangaroo Bot | Kangaroo LLM | training | 1.5% | 437 of 29,982 | level |
| Meta-WebIndexer | Meta | answer index | 1.4% | 407 of 29,982 | level |
| Applebot | Apple | answer index | 1.2% | 351 of 29,982 | level |
| Amzn-SearchBot | Amazon | answer index | 0.9% | 255 of 29,982 | level |
| TikTokSpider | ByteDance | live retrieval | 0.8% | 246 of 29,982 | level |
| Amzn-User | Amazon | live retrieval | 0.7% | 209 of 29,982 | level |
| MistralAI-Index | Mistral AI | answer index | 0.4% | 112 of 29,982 | level |
| facebookexternalhit | Meta | live retrieval | 0.2% | 56 of 29,982 | level |
| Googlebot-News | answer index | 0.1% | 39 of 29,982 | level | |
| Meta-ExternalAds | Meta | training | 0.1% | 39 of 29,982 | level |
| Bingbot | Microsoft | answer index | 0.1% | 35 of 29,982 | level |
| msnbot | Microsoft | answer index | 0.1% | 18 of 29,982 | level |
| Googlebot | answer index | 0.1% | 17 of 29,982 | level | |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 14 of 29,982 | level |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 200 total
- robinhood.com Published an llms.txt file
- kir2kos.net Published an llms.txt file
- nexon.com AI access score moved 59 to 49
- bgtee.com GPTBot is now blocked by robots.txt
- bgtee.com ClaudeBot is now blocked by robots.txt
- bgtee.com anthropic-ai is now blocked by robots.txt
- bgtee.com Google-Extended is now blocked by robots.txt
- bgtee.com Google-CloudVertexBot is now blocked by robots.txt
- bgtee.com GoogleOther is now blocked by robots.txt
- bgtee.com Applebot-Extended is now blocked by robots.txt
- bgtee.com PerplexityBot is now refused at the network edge, whatever robots.txt permits
- bgtee.com Meta-ExternalAgent is now blocked by robots.txt
- bgtee.com Bytespider is now blocked by robots.txt
- bgtee.com Amazonbot is now blocked by robots.txt
- bgtee.com CCBot is now blocked by robots.txt
- bgtee.com Diffbot is now blocked by robots.txt
- bgtee.com omgili is now blocked by robots.txt
- bgtee.com omgilibot is now blocked by robots.txt
- bgtee.com cohere-ai is now blocked by robots.txt
- bgtee.com MistralAI-Training is now blocked by robots.txt
- mamastar.jp AI access score moved 74 to 49
- launchpad.net OAI-SearchBot is no longer refused at the network edge
- launchpad.net ClaudeBot is no longer refused at the network edge
- launchpad.net PerplexityBot is no longer refused at the network edge
- launchpad.net AI access score moved 39 to 52
- iea.org AI access score moved 73 to 88
- imageshack.us MistralAI-Training is now blocked by robots.txt
- tn.gov.in AI access score moved 65 to 46
- rajasthan.gov.in Published an llms.txt file
- square.site ClaudeBot is now refused at the network edge, whatever robots.txt permits
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2026-W46", 2026-11-15. https://crawlcensus.com/report/2026-W46
Machine-readable: census totals, full corpus, crawler registry. Method: methodology.