Weekly report · 2026-W43

The state of AI crawling

Measured on 29,997 reachable domains for the week ending 2026-10-25. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.

← 2026-W42  ·  2026-W44 →

29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt

Block rate by crawler with change on the previous week

CrawlerOperatorContent used forBlock rateSitesWeek on week
GPTBot OpenAI training 7.9% 2,377 of 30,253 +0.7 pt
CCBot Common Crawl Foundation archive 7.5% 2,277 of 30,253 +0.7 pt
Bytespider ByteDance training 7.3% 2,207 of 30,253 +0.8 pt
ClaudeBot Anthropic training 6.6% 1,985 of 30,253 +0.7 pt
Meta-ExternalAgent Meta training 5.6% 1,700 of 30,253 +0.9 pt
Amazonbot Amazon training 5.6% 1,684 of 30,253 +0.8 pt
Applebot-Extended Apple training 5.6% 1,683 of 30,253 +0.8 pt
Google-Extended Google training 5.5% 1,676 of 30,253 +0.8 pt
anthropic-ai Anthropic training 5.2% 1,585 of 30,253 +0.8 pt
omgilibot Webz.io training 5.2% 1,573 of 30,253 +0.8 pt
Diffbot Diffbot training 5.0% 1,520 of 30,253 +0.8 pt
cohere-ai Cohere live retrieval 5.0% 1,508 of 30,253 +0.8 pt
omgili Webz.io training 4.8% 1,460 of 30,253 +0.8 pt
SemrushBot-OCOB Semrush training 3.5% 1,060 of 30,253 -0.2 pt
PerplexityBot Perplexity answer index 3.4% 1,023 of 30,253 level
ChatGPT-User OpenAI live retrieval 3.3% 1,008 of 30,253 -0.1 pt
FacebookBot Meta training 3.1% 940 of 30,253 -0.1 pt
YouBot You.com answer index 3.0% 895 of 30,253 level
ImagesiftBot ImageSift (Hive) training 2.7% 825 of 30,253 level
Timpibot Timpi training 2.7% 825 of 30,253 level
Google-CloudVertexBot Google live retrieval 2.4% 726 of 30,253 +0.8 pt
Scrapy Zyte (open-source framework) training 2.4% 721 of 30,253 level
AI2Bot Allen Institute for AI training 2.3% 705 of 30,253 level
GoogleOther Google training 2.3% 689 of 30,253 +0.8 pt
OAI-SearchBot OpenAI answer index 2.1% 623 of 30,253 level
cohere-training-data-crawler Cohere training 2.0% 611 of 30,253 level
Meta-ExternalFetcher Meta live retrieval 1.8% 532 of 30,253 level
DuckAssistBot DuckDuckGo live retrieval 1.7% 518 of 30,253 level
PanguBot Huawei training 1.7% 513 of 30,253 level
Claude-User Anthropic live retrieval 1.7% 500 of 30,253 level
MistralAI-Training Mistral AI training 1.6% 480 of 30,253 +0.9 pt
Perplexity-User Perplexity live retrieval 1.6% 476 of 30,253 level
Claude-SearchBot Anthropic answer index 1.6% 469 of 30,253 level
MistralAI-User Mistral AI live retrieval 1.5% 465 of 30,253 level
Kangaroo Bot Kangaroo LLM training 1.5% 453 of 30,253 level
Meta-WebIndexer Meta answer index 1.2% 363 of 30,253 level
Applebot Apple answer index 1.2% 354 of 30,253 level
TikTokSpider ByteDance live retrieval 0.8% 253 of 30,253 level
Amzn-SearchBot Amazon answer index 0.8% 248 of 30,253 level
Amzn-User Amazon live retrieval 0.6% 194 of 30,253 level
MistralAI-Index Mistral AI answer index 0.3% 97 of 30,253 level
facebookexternalhit Meta live retrieval 0.2% 53 of 30,253 level
Googlebot-News Google answer index 0.1% 35 of 30,253 level
Bingbot Microsoft answer index 0.1% 33 of 30,253 level
Meta-ExternalAds Meta training 0.1% 26 of 30,253 level
Googlebot Google answer index 0.1% 17 of 30,253 level
msnbot Microsoft answer index 0.1% 17 of 30,253 level
OAI-AdsBot OpenAI live retrieval 0.0% 12 of 30,253 level

Trend share of measured sites

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Where the blocking is share blocking an answer engine, by top-level domain

bar chart.uk (n=429)17.5%.fr (n=320)16.6%.de (n=823)16.5%.com (n=14603)9.8%.org (n=1840)9.2%.jp (n=401)9.2%.net (n=1306)9%.br (n=360)7.5%.in (n=302)5.6%.io (n=654)3.8%.edu (n=471)3%.ru (n=1157)2.9%

Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.

Changes recorded this week 200 total

Full change log · RSS

Cite this CC BY 4.0

The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.

Crawl Census, "The state of AI crawling, 2026-W43", 2026-10-25. https://crawlcensus.com/report/2026-W43

Machine-readable: census totals, full corpus, crawler registry. Method: methodology.