Weekly report · 2026-W40
The state of AI crawling
Measured on 30,225 reachable domains for the week ending 2026-10-04. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
30,225
domains measured
7%
block an answer engine
19%
refuse AI agents at the edge
14%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 7.1% | 2,143 of 30,170 | +0.2 pt |
| CCBot | Common Crawl Foundation | archive | 6.9% | 2,068 of 30,170 | +0.1 pt |
| Bytespider | ByteDance | training | 6.5% | 1,947 of 30,170 | +0.1 pt |
| ClaudeBot | Anthropic | training | 5.8% | 1,746 of 30,170 | +0.2 pt |
| Meta-ExternalAgent | Meta | training | 4.7% | 1,423 of 30,170 | +0.2 pt |
| Amazonbot | Amazon | training | 4.7% | 1,408 of 30,170 | +0.1 pt |
| Google-Extended | training | 4.6% | 1,389 of 30,170 | +0.1 pt | |
| Applebot-Extended | Apple | training | 4.6% | 1,376 of 30,170 | +0.2 pt |
| anthropic-ai | Anthropic | training | 4.4% | 1,318 of 30,170 | +0.1 pt |
| omgilibot | Webz.io | training | 4.3% | 1,286 of 30,170 | level |
| cohere-ai | Cohere | live retrieval | 4.1% | 1,223 of 30,170 | +0.1 pt |
| Diffbot | Diffbot | training | 4.0% | 1,219 of 30,170 | +0.1 pt |
| omgili | Webz.io | training | 3.9% | 1,170 of 30,170 | level |
| SemrushBot-OCOB | Semrush | training | 3.7% | 1,122 of 30,170 | +0.1 pt |
| ChatGPT-User | OpenAI | live retrieval | 3.5% | 1,052 of 30,170 | level |
| PerplexityBot | Perplexity | answer index | 3.5% | 1,051 of 30,170 | level |
| FacebookBot | Meta | training | 3.2% | 975 of 30,170 | +0.1 pt |
| YouBot | You.com | answer index | 2.9% | 889 of 30,170 | level |
| ImagesiftBot | ImageSift (Hive) | training | 2.8% | 851 of 30,170 | +0.1 pt |
| Timpibot | Timpi | training | 2.7% | 829 of 30,170 | +0.1 pt |
| Scrapy | Zyte (open-source framework) | training | 2.5% | 745 of 30,170 | +0.1 pt |
| AI2Bot | Allen Institute for AI | training | 2.3% | 698 of 30,170 | +0.1 pt |
| OAI-SearchBot | OpenAI | answer index | 2.1% | 643 of 30,170 | level |
| cohere-training-data-crawler | Cohere | training | 2.0% | 614 of 30,170 | level |
| Meta-ExternalFetcher | Meta | live retrieval | 1.8% | 556 of 30,170 | +0.1 pt |
| DuckAssistBot | DuckDuckGo | live retrieval | 1.7% | 513 of 30,170 | level |
| PanguBot | Huawei | training | 1.7% | 513 of 30,170 | +0.1 pt |
| Claude-User | Anthropic | live retrieval | 1.6% | 490 of 30,170 | level |
| Claude-SearchBot | Anthropic | answer index | 1.6% | 472 of 30,170 | level |
| Kangaroo Bot | Kangaroo LLM | training | 1.5% | 463 of 30,170 | level |
| Perplexity-User | Perplexity | live retrieval | 1.5% | 461 of 30,170 | level |
| MistralAI-User | Mistral AI | live retrieval | 1.5% | 439 of 30,170 | level |
| Google-CloudVertexBot | live retrieval | 1.4% | 418 of 30,170 | +0.1 pt | |
| GoogleOther | training | 1.3% | 379 of 30,170 | +0.1 pt | |
| Meta-WebIndexer | Meta | answer index | 1.2% | 358 of 30,170 | level |
| Applebot | Apple | answer index | 1.1% | 339 of 30,170 | level |
| TikTokSpider | ByteDance | live retrieval | 0.8% | 251 of 30,170 | level |
| Amzn-SearchBot | Amazon | answer index | 0.7% | 226 of 30,170 | level |
| Amzn-User | Amazon | live retrieval | 0.5% | 160 of 30,170 | level |
| MistralAI-Training | Mistral AI | training | 0.5% | 137 of 30,170 | +0.1 pt |
| MistralAI-Index | Mistral AI | answer index | 0.2% | 75 of 30,170 | level |
| facebookexternalhit | Meta | live retrieval | 0.2% | 53 of 30,170 | level |
| Bingbot | Microsoft | answer index | 0.1% | 40 of 30,170 | level |
| Googlebot-News | answer index | 0.1% | 36 of 30,170 | level | |
| Meta-ExternalAds | Meta | training | 0.1% | 22 of 30,170 | level |
| msnbot | Microsoft | answer index | 0.1% | 21 of 30,170 | level |
| Googlebot | answer index | 0.1% | 17 of 30,170 | level | |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 10 of 30,170 | level |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 200 total
- nrc.ca ClaudeBot is now refused at the network edge, whatever robots.txt permits
- pmkisan.gov.in AI access score moved 66 to 46
- u.gg GPTBot is now blocked by robots.txt
- u.gg GPTBot is now refused at the network edge, whatever robots.txt permits
- u.gg OAI-SearchBot is now refused at the network edge, whatever robots.txt permits
- u.gg ClaudeBot is now blocked by robots.txt
- u.gg ClaudeBot is now refused at the network edge, whatever robots.txt permits
- u.gg anthropic-ai is now blocked by robots.txt
- u.gg Google-Extended is now blocked by robots.txt
- u.gg Google-CloudVertexBot is now blocked by robots.txt
- u.gg GoogleOther is now blocked by robots.txt
- u.gg Applebot-Extended is now blocked by robots.txt
- u.gg PerplexityBot is now refused at the network edge, whatever robots.txt permits
- u.gg Meta-ExternalAgent is now blocked by robots.txt
- u.gg Bytespider is now blocked by robots.txt
- u.gg Amazonbot is now blocked by robots.txt
- u.gg CCBot is now blocked by robots.txt
- u.gg Diffbot is now blocked by robots.txt
- u.gg omgili is now blocked by robots.txt
- u.gg omgilibot is now blocked by robots.txt
- u.gg cohere-ai is now blocked by robots.txt
- u.gg MistralAI-Training is now blocked by robots.txt
- cinelab.ru AI access score moved 65 to 47
- gem.gov.in AI access score moved 75 to 46
- tapology.com GPTBot is now blocked by robots.txt
- tapology.com ClaudeBot is now blocked by robots.txt
- tapology.com anthropic-ai is now blocked by robots.txt
- tapology.com Google-Extended is now blocked by robots.txt
- tapology.com PerplexityBot is now blocked by robots.txt
- tapology.com Meta-ExternalAgent is now blocked by robots.txt
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2026-W40", 2026-10-04. https://crawlcensus.com/report/2026-W40Machine-readable: census totals, full corpus, crawler registry. Method: methodology.