Weekly report · 2026-W42
The state of AI crawling
Measured on 29,997 reachable domains for the week ending 2026-10-18. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 7.1% | 2,169 of 30,399 | level |
| CCBot | Common Crawl Foundation | archive | 6.8% | 2,062 of 30,399 | level |
| Bytespider | ByteDance | training | 6.5% | 1,981 of 30,399 | level |
| ClaudeBot | Anthropic | training | 5.8% | 1,767 of 30,399 | level |
| Amazonbot | Amazon | training | 4.8% | 1,461 of 30,399 | +0.1 pt |
| Applebot-Extended | Apple | training | 4.8% | 1,447 of 30,399 | +0.2 pt |
| Meta-ExternalAgent | Meta | training | 4.8% | 1,446 of 30,399 | level |
| Google-Extended | training | 4.7% | 1,435 of 30,399 | +0.1 pt | |
| anthropic-ai | Anthropic | training | 4.4% | 1,346 of 30,399 | level |
| omgilibot | Webz.io | training | 4.4% | 1,342 of 30,399 | +0.1 pt |
| Diffbot | Diffbot | training | 4.2% | 1,272 of 30,399 | +0.1 pt |
| cohere-ai | Cohere | live retrieval | 4.1% | 1,259 of 30,399 | +0.1 pt |
| omgili | Webz.io | training | 4.0% | 1,230 of 30,399 | +0.1 pt |
| SemrushBot-OCOB | Semrush | training | 3.7% | 1,124 of 30,399 | -0.1 pt |
| ChatGPT-User | OpenAI | live retrieval | 3.4% | 1,036 of 30,399 | -0.1 pt |
| PerplexityBot | Perplexity | answer index | 3.4% | 1,034 of 30,399 | -0.1 pt |
| FacebookBot | Meta | training | 3.2% | 964 of 30,399 | -0.1 pt |
| YouBot | You.com | answer index | 3.0% | 898 of 30,399 | -0.1 pt |
| ImagesiftBot | ImageSift (Hive) | training | 2.8% | 841 of 30,399 | level |
| Timpibot | Timpi | training | 2.7% | 826 of 30,399 | level |
| Scrapy | Zyte (open-source framework) | training | 2.4% | 727 of 30,399 | level |
| AI2Bot | Allen Institute for AI | training | 2.3% | 702 of 30,399 | level |
| OAI-SearchBot | OpenAI | answer index | 2.1% | 629 of 30,399 | -0.1 pt |
| cohere-training-data-crawler | Cohere | training | 2.0% | 617 of 30,399 | level |
| Meta-ExternalFetcher | Meta | live retrieval | 1.7% | 530 of 30,399 | level |
| DuckAssistBot | DuckDuckGo | live retrieval | 1.7% | 525 of 30,399 | level |
| PanguBot | Huawei | training | 1.7% | 518 of 30,399 | level |
| Claude-User | Anthropic | live retrieval | 1.6% | 498 of 30,399 | level |
| Claude-SearchBot | Anthropic | answer index | 1.6% | 477 of 30,399 | level |
| Google-CloudVertexBot | live retrieval | 1.6% | 476 of 30,399 | +0.1 pt | |
| Perplexity-User | Perplexity | live retrieval | 1.6% | 474 of 30,399 | level |
| MistralAI-User | Mistral AI | live retrieval | 1.5% | 465 of 30,399 | level |
| Kangaroo Bot | Kangaroo LLM | training | 1.5% | 452 of 30,399 | level |
| GoogleOther | training | 1.4% | 434 of 30,399 | +0.1 pt | |
| Meta-WebIndexer | Meta | answer index | 1.2% | 361 of 30,399 | level |
| Applebot | Apple | answer index | 1.2% | 353 of 30,399 | level |
| TikTokSpider | ByteDance | live retrieval | 0.9% | 260 of 30,399 | level |
| Amzn-SearchBot | Amazon | answer index | 0.8% | 242 of 30,399 | level |
| MistralAI-Training | Mistral AI | training | 0.7% | 205 of 30,399 | +0.1 pt |
| Amzn-User | Amazon | live retrieval | 0.6% | 182 of 30,399 | level |
| MistralAI-Index | Mistral AI | answer index | 0.3% | 93 of 30,399 | level |
| facebookexternalhit | Meta | live retrieval | 0.2% | 51 of 30,399 | level |
| Bingbot | Microsoft | answer index | 0.1% | 38 of 30,399 | level |
| Googlebot-News | answer index | 0.1% | 36 of 30,399 | level | |
| Meta-ExternalAds | Meta | training | 0.1% | 24 of 30,399 | level |
| msnbot | Microsoft | answer index | 0.1% | 19 of 30,399 | level |
| Googlebot | answer index | 0.0% | 15 of 30,399 | level | |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 12 of 30,399 | level |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 200 total
- pik.ru Published an llms.txt file
- biorxiv.org PerplexityBot is now refused at the network edge, whatever robots.txt permits
- biorxiv.org AI access score moved 48 to 58
- ubs.com AI access score moved 62 to 96
- lixiang.com AI access score moved 56 to 46
- ualberta.ca AI access score moved 77 to 47
- defense.gov GPTBot is now refused at the network edge, whatever robots.txt permits
- defense.gov OAI-SearchBot is now refused at the network edge, whatever robots.txt permits
- defense.gov ClaudeBot is now refused at the network edge, whatever robots.txt permits
- defense.gov PerplexityBot is now refused at the network edge, whatever robots.txt permits
- defense.gov AI access score moved 54 to 67
- bcg.com GPTBot is now refused at the network edge, whatever robots.txt permits
- bcg.com OAI-SearchBot is now refused at the network edge, whatever robots.txt permits
- bcg.com ClaudeBot is now refused at the network edge, whatever robots.txt permits
- bcg.com PerplexityBot is now refused at the network edge, whatever robots.txt permits
- bcg.com AI access score moved 58 to 75
- lesechos.fr GPTBot is now blocked by robots.txt
- lesechos.fr GPTBot is now refused at the network edge, whatever robots.txt permits
- lesechos.fr OAI-SearchBot is now blocked by robots.txt
- lesechos.fr OAI-SearchBot is now refused at the network edge, whatever robots.txt permits
- lesechos.fr ChatGPT-User is now blocked by robots.txt
- lesechos.fr ClaudeBot is now blocked by robots.txt
- lesechos.fr ClaudeBot is now refused at the network edge, whatever robots.txt permits
- lesechos.fr Claude-User is now blocked by robots.txt
- lesechos.fr Claude-SearchBot is now blocked by robots.txt
- lesechos.fr anthropic-ai is now blocked by robots.txt
- lesechos.fr Google-Extended is now blocked by robots.txt
- lesechos.fr Applebot is now blocked by robots.txt
- lesechos.fr Applebot-Extended is now blocked by robots.txt
- lesechos.fr PerplexityBot is now blocked by robots.txt
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2026-W42", 2026-10-18. https://crawlcensus.com/report/2026-W42
Machine-readable: census totals, full corpus, crawler registry. Method: methodology.