Weekly report · 2026-W44
The state of AI crawling
Measured on 29,997 reachable domains for the week ending 2026-11-01. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 9.6% | 2,889 of 29,997 | +1.8 pt |
| CCBot | Common Crawl Foundation | archive | 9.4% | 2,829 of 29,997 | +1.9 pt |
| Bytespider | ByteDance | training | 9.2% | 2,760 of 29,997 | +1.9 pt |
| ClaudeBot | Anthropic | training | 8.4% | 2,524 of 29,997 | +1.9 pt |
| Meta-ExternalAgent | Meta | training | 7.6% | 2,282 of 29,997 | +2.0 pt |
| Amazonbot | Amazon | training | 7.5% | 2,246 of 29,997 | +1.9 pt |
| Applebot-Extended | Apple | training | 7.4% | 2,231 of 29,997 | +1.9 pt |
| Google-Extended | training | 7.4% | 2,222 of 29,997 | +1.9 pt | |
| anthropic-ai | Anthropic | training | 7.1% | 2,138 of 29,997 | +1.9 pt |
| omgilibot | Webz.io | training | 7.1% | 2,117 of 29,997 | +1.9 pt |
| Diffbot | Diffbot | training | 6.9% | 2,070 of 29,997 | +1.9 pt |
| cohere-ai | Cohere | live retrieval | 6.9% | 2,062 of 29,997 | +1.9 pt |
| omgili | Webz.io | training | 6.7% | 2,004 of 29,997 | +1.9 pt |
| Google-CloudVertexBot | live retrieval | 4.3% | 1,304 of 29,997 | +1.9 pt | |
| GoogleOther | training | 4.2% | 1,263 of 29,997 | +1.9 pt | |
| MistralAI-Training | Mistral AI | training | 3.6% | 1,069 of 29,997 | +2.0 pt |
| SemrushBot-OCOB | Semrush | training | 3.5% | 1,056 of 29,997 | level |
| PerplexityBot | Perplexity | answer index | 3.4% | 1,019 of 29,997 | level |
| ChatGPT-User | OpenAI | live retrieval | 3.3% | 986 of 29,997 | level |
| FacebookBot | Meta | training | 3.1% | 923 of 29,997 | level |
| YouBot | You.com | answer index | 2.9% | 883 of 29,997 | level |
| Timpibot | Timpi | training | 2.7% | 817 of 29,997 | level |
| ImagesiftBot | ImageSift (Hive) | training | 2.7% | 812 of 29,997 | level |
| Scrapy | Zyte (open-source framework) | training | 2.4% | 713 of 29,997 | level |
| AI2Bot | Allen Institute for AI | training | 2.3% | 694 of 29,997 | level |
| cohere-training-data-crawler | Cohere | training | 2.0% | 614 of 29,997 | level |
| OAI-SearchBot | OpenAI | answer index | 2.0% | 612 of 29,997 | level |
| Meta-ExternalFetcher | Meta | live retrieval | 1.8% | 551 of 29,997 | +0.1 pt |
| PanguBot | Huawei | training | 1.7% | 504 of 29,997 | level |
| DuckAssistBot | DuckDuckGo | live retrieval | 1.7% | 498 of 29,997 | -0.1 pt |
| Claude-User | Anthropic | live retrieval | 1.7% | 496 of 29,997 | level |
| Perplexity-User | Perplexity | live retrieval | 1.6% | 477 of 29,997 | level |
| Claude-SearchBot | Anthropic | answer index | 1.5% | 461 of 29,997 | level |
| MistralAI-User | Mistral AI | live retrieval | 1.5% | 456 of 29,997 | level |
| Kangaroo Bot | Kangaroo LLM | training | 1.5% | 437 of 29,997 | level |
| Meta-WebIndexer | Meta | answer index | 1.3% | 399 of 29,997 | +0.1 pt |
| Applebot | Apple | answer index | 1.2% | 355 of 29,997 | level |
| Amzn-SearchBot | Amazon | answer index | 0.9% | 259 of 29,997 | level |
| TikTokSpider | ByteDance | live retrieval | 0.8% | 247 of 29,997 | level |
| Amzn-User | Amazon | live retrieval | 0.7% | 206 of 29,997 | level |
| MistralAI-Index | Mistral AI | answer index | 0.4% | 111 of 29,997 | level |
| facebookexternalhit | Meta | live retrieval | 0.2% | 56 of 29,997 | level |
| Googlebot-News | answer index | 0.1% | 40 of 29,997 | level | |
| Meta-ExternalAds | Meta | training | 0.1% | 39 of 29,997 | level |
| Bingbot | Microsoft | answer index | 0.1% | 35 of 29,997 | level |
| Googlebot | answer index | 0.1% | 19 of 29,997 | level | |
| msnbot | Microsoft | answer index | 0.1% | 19 of 29,997 | level |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 14 of 29,997 | level |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 200 total
- gaemt.ru SemrushBot-OCOB is now allowed by robots.txt
- gaemt.ru AI access score moved 75 to 46
- inist.fr AI access score moved 56 to 92
- xn--72czpfe7gxb9cveycua.com GPTBot is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com ClaudeBot is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com anthropic-ai is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com Google-Extended is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com Google-CloudVertexBot is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com GoogleOther is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com Applebot-Extended is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com Meta-ExternalAgent is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com Bytespider is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com Amazonbot is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com CCBot is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com Diffbot is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com omgili is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com omgilibot is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com cohere-ai is now blocked by robots.txt
- xn--72czpfe7gxb9cveycua.com MistralAI-Training is now blocked by robots.txt
- rulive.ru GPTBot is now refused at the network edge, whatever robots.txt permits
- rulive.ru OAI-SearchBot is now refused at the network edge, whatever robots.txt permits
- plumgoodness.com Removed its llms.txt file
- plumgoodness.com AI access score moved 90 to 53
- mamen123ai.com GPTBot is now allowed by robots.txt
- mamen123ai.com ClaudeBot is now allowed by robots.txt
- mamen123ai.com anthropic-ai is now allowed by robots.txt
- mamen123ai.com Google-Extended is now allowed by robots.txt
- mamen123ai.com Google-CloudVertexBot is now allowed by robots.txt
- mamen123ai.com GoogleOther is now allowed by robots.txt
- mamen123ai.com Applebot-Extended is now allowed by robots.txt
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2026-W44", 2026-11-01. https://crawlcensus.com/report/2026-W44
Machine-readable: census totals, full corpus, crawler registry. Method: methodology.