Weekly report · 2026-W44

The state of AI crawling

Measured on 29,997 reachable domains for the week ending 2026-11-01. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.

← 2026-W43  ·  2026-W45 →

29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt

Block rate by crawler with change on the previous week

CrawlerOperatorContent used forBlock rateSitesWeek on week
GPTBot OpenAI training 9.6% 2,889 of 29,997 +1.8 pt
CCBot Common Crawl Foundation archive 9.4% 2,829 of 29,997 +1.9 pt
Bytespider ByteDance training 9.2% 2,760 of 29,997 +1.9 pt
ClaudeBot Anthropic training 8.4% 2,524 of 29,997 +1.9 pt
Meta-ExternalAgent Meta training 7.6% 2,282 of 29,997 +2.0 pt
Amazonbot Amazon training 7.5% 2,246 of 29,997 +1.9 pt
Applebot-Extended Apple training 7.4% 2,231 of 29,997 +1.9 pt
Google-Extended Google training 7.4% 2,222 of 29,997 +1.9 pt
anthropic-ai Anthropic training 7.1% 2,138 of 29,997 +1.9 pt
omgilibot Webz.io training 7.1% 2,117 of 29,997 +1.9 pt
Diffbot Diffbot training 6.9% 2,070 of 29,997 +1.9 pt
cohere-ai Cohere live retrieval 6.9% 2,062 of 29,997 +1.9 pt
omgili Webz.io training 6.7% 2,004 of 29,997 +1.9 pt
Google-CloudVertexBot Google live retrieval 4.3% 1,304 of 29,997 +1.9 pt
GoogleOther Google training 4.2% 1,263 of 29,997 +1.9 pt
MistralAI-Training Mistral AI training 3.6% 1,069 of 29,997 +2.0 pt
SemrushBot-OCOB Semrush training 3.5% 1,056 of 29,997 level
PerplexityBot Perplexity answer index 3.4% 1,019 of 29,997 level
ChatGPT-User OpenAI live retrieval 3.3% 986 of 29,997 level
FacebookBot Meta training 3.1% 923 of 29,997 level
YouBot You.com answer index 2.9% 883 of 29,997 level
Timpibot Timpi training 2.7% 817 of 29,997 level
ImagesiftBot ImageSift (Hive) training 2.7% 812 of 29,997 level
Scrapy Zyte (open-source framework) training 2.4% 713 of 29,997 level
AI2Bot Allen Institute for AI training 2.3% 694 of 29,997 level
cohere-training-data-crawler Cohere training 2.0% 614 of 29,997 level
OAI-SearchBot OpenAI answer index 2.0% 612 of 29,997 level
Meta-ExternalFetcher Meta live retrieval 1.8% 551 of 29,997 +0.1 pt
PanguBot Huawei training 1.7% 504 of 29,997 level
DuckAssistBot DuckDuckGo live retrieval 1.7% 498 of 29,997 -0.1 pt
Claude-User Anthropic live retrieval 1.7% 496 of 29,997 level
Perplexity-User Perplexity live retrieval 1.6% 477 of 29,997 level
Claude-SearchBot Anthropic answer index 1.5% 461 of 29,997 level
MistralAI-User Mistral AI live retrieval 1.5% 456 of 29,997 level
Kangaroo Bot Kangaroo LLM training 1.5% 437 of 29,997 level
Meta-WebIndexer Meta answer index 1.3% 399 of 29,997 +0.1 pt
Applebot Apple answer index 1.2% 355 of 29,997 level
Amzn-SearchBot Amazon answer index 0.9% 259 of 29,997 level
TikTokSpider ByteDance live retrieval 0.8% 247 of 29,997 level
Amzn-User Amazon live retrieval 0.7% 206 of 29,997 level
MistralAI-Index Mistral AI answer index 0.4% 111 of 29,997 level
facebookexternalhit Meta live retrieval 0.2% 56 of 29,997 level
Googlebot-News Google answer index 0.1% 40 of 29,997 level
Meta-ExternalAds Meta training 0.1% 39 of 29,997 level
Bingbot Microsoft answer index 0.1% 35 of 29,997 level
Googlebot Google answer index 0.1% 19 of 29,997 level
msnbot Microsoft answer index 0.1% 19 of 29,997 level
OAI-AdsBot OpenAI live retrieval 0.0% 14 of 29,997 level

Trend share of measured sites

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Where the blocking is share blocking an answer engine, by top-level domain

bar chart.uk (n=429)17.5%.fr (n=320)16.6%.de (n=823)16.5%.com (n=14603)9.8%.org (n=1840)9.2%.jp (n=401)9.2%.net (n=1306)9%.br (n=360)7.5%.in (n=302)5.6%.io (n=654)3.8%.edu (n=471)3%.ru (n=1157)2.9%

Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.

Changes recorded this week 200 total

Full change log · RSS

Cite this CC BY 4.0

The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.

Crawl Census, "The state of AI crawling, 2026-W44", 2026-11-01. https://crawlcensus.com/report/2026-W44

Machine-readable: census totals, full corpus, crawler registry. Method: methodology.