Weekly report · 2026-W41

The state of AI crawling

Measured on 29,997 reachable domains for the week ending 2026-10-11. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.

← 2026-W40  ·  2026-W42 →

29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt

Block rate by crawler with change on the previous week

CrawlerOperatorContent used forBlock rateSitesWeek on week
GPTBot OpenAI training 7.1% 2,158 of 30,292 level
CCBot Common Crawl Foundation archive 6.8% 2,055 of 30,292 -0.1 pt
Bytespider ByteDance training 6.5% 1,965 of 30,292 level
ClaudeBot Anthropic training 5.8% 1,746 of 30,292 level
Meta-ExternalAgent Meta training 4.7% 1,436 of 30,292 level
Amazonbot Amazon training 4.7% 1,419 of 30,292 level
Google-Extended Google training 4.6% 1,398 of 30,292 level
Applebot-Extended Apple training 4.6% 1,393 of 30,292 level
anthropic-ai Anthropic training 4.4% 1,338 of 30,292 level
omgilibot Webz.io training 4.4% 1,318 of 30,292 +0.1 pt
Diffbot Diffbot training 4.1% 1,245 of 30,292 +0.1 pt
cohere-ai Cohere live retrieval 4.1% 1,237 of 30,292 level
omgili Webz.io training 4.0% 1,205 of 30,292 +0.1 pt
SemrushBot-OCOB Semrush training 3.8% 1,136 of 30,292 level
ChatGPT-User OpenAI live retrieval 3.5% 1,059 of 30,292 level
PerplexityBot Perplexity answer index 3.5% 1,059 of 30,292 level
FacebookBot Meta training 3.2% 979 of 30,292 level
YouBot You.com answer index 3.0% 911 of 30,292 +0.1 pt
ImagesiftBot ImageSift (Hive) training 2.8% 848 of 30,292 level
Timpibot Timpi training 2.8% 836 of 30,292 level
Scrapy Zyte (open-source framework) training 2.4% 737 of 30,292 level
AI2Bot Allen Institute for AI training 2.3% 705 of 30,292 level
OAI-SearchBot OpenAI answer index 2.1% 646 of 30,292 level
cohere-training-data-crawler Cohere training 2.1% 622 of 30,292 level
Meta-ExternalFetcher Meta live retrieval 1.8% 542 of 30,292 -0.1 pt
DuckAssistBot DuckDuckGo live retrieval 1.7% 527 of 30,292 level
PanguBot Huawei training 1.7% 520 of 30,292 level
Claude-User Anthropic live retrieval 1.7% 500 of 30,292 level
Claude-SearchBot Anthropic answer index 1.6% 488 of 30,292 level
Perplexity-User Perplexity live retrieval 1.6% 476 of 30,292 level
MistralAI-User Mistral AI live retrieval 1.5% 464 of 30,292 +0.1 pt
Kangaroo Bot Kangaroo LLM training 1.5% 458 of 30,292 level
Google-CloudVertexBot Google live retrieval 1.4% 433 of 30,292 level
GoogleOther Google training 1.3% 396 of 30,292 +0.1 pt
Meta-WebIndexer Meta answer index 1.2% 360 of 30,292 level
Applebot Apple answer index 1.1% 337 of 30,292 level
TikTokSpider ByteDance live retrieval 0.9% 260 of 30,292 level
Amzn-SearchBot Amazon answer index 0.8% 244 of 30,292 +0.1 pt
Amzn-User Amazon live retrieval 0.6% 178 of 30,292 +0.1 pt
MistralAI-Training Mistral AI training 0.5% 160 of 30,292 +0.1 pt
MistralAI-Index Mistral AI answer index 0.3% 93 of 30,292 +0.1 pt
facebookexternalhit Meta live retrieval 0.2% 55 of 30,292 level
Bingbot Microsoft answer index 0.1% 38 of 30,292 level
Googlebot-News Google answer index 0.1% 38 of 30,292 level
Meta-ExternalAds Meta training 0.1% 21 of 30,292 level
msnbot Microsoft answer index 0.1% 20 of 30,292 level
Googlebot Google answer index 0.1% 16 of 30,292 level
OAI-AdsBot OpenAI live retrieval 0.0% 11 of 30,292 level

Trend share of measured sites

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Where the blocking is share blocking an answer engine, by top-level domain

bar chart.uk (n=429)17.5%.fr (n=320)16.6%.de (n=823)16.5%.com (n=14603)9.8%.org (n=1840)9.2%.jp (n=401)9.2%.net (n=1306)9%.br (n=360)7.5%.in (n=302)5.6%.io (n=654)3.8%.edu (n=471)3%.ru (n=1157)2.9%

Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.

Changes recorded this week 200 total

Full change log · RSS

Cite this CC BY 4.0

The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.

Crawl Census, "The state of AI crawling, 2026-W41", 2026-10-11. https://crawlcensus.com/report/2026-W41

Machine-readable: census totals, full corpus, crawler registry. Method: methodology.