Weekly report · 2026-W45

The state of AI crawling

Measured on 29,997 reachable domains for the week ending 2026-11-08. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.

← 2026-W44  ·  2026-W46 →

29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt

Block rate by crawler with change on the previous week

CrawlerOperatorContent used forBlock rateSitesWeek on week
GPTBot OpenAI training 9.8% 2,931 of 29,967 +0.1 pt
CCBot Common Crawl Foundation archive 9.6% 2,870 of 29,967 +0.1 pt
Bytespider ByteDance training 9.4% 2,804 of 29,967 +0.2 pt
ClaudeBot Anthropic training 8.6% 2,568 of 29,967 +0.2 pt
Meta-ExternalAgent Meta training 7.8% 2,334 of 29,967 +0.2 pt
Amazonbot Amazon training 7.6% 2,283 of 29,967 +0.1 pt
Applebot-Extended Apple training 7.6% 2,280 of 29,967 +0.2 pt
Google-Extended Google training 7.6% 2,263 of 29,967 +0.1 pt
anthropic-ai Anthropic training 7.3% 2,183 of 29,967 +0.2 pt
omgilibot Webz.io training 7.2% 2,160 of 29,967 +0.2 pt
Diffbot Diffbot training 7.1% 2,115 of 29,967 +0.2 pt
cohere-ai Cohere live retrieval 7.0% 2,109 of 29,967 +0.2 pt
omgili Webz.io training 6.8% 2,051 of 29,967 +0.2 pt
Google-CloudVertexBot Google live retrieval 4.5% 1,356 of 29,967 +0.2 pt
GoogleOther Google training 4.4% 1,308 of 29,967 +0.2 pt
MistralAI-Training Mistral AI training 3.7% 1,117 of 29,967 +0.2 pt
SemrushBot-OCOB Semrush training 3.5% 1,062 of 29,967 level
PerplexityBot Perplexity answer index 3.4% 1,013 of 29,967 level
ChatGPT-User OpenAI live retrieval 3.3% 990 of 29,967 level
FacebookBot Meta training 3.1% 923 of 29,967 level
YouBot You.com answer index 2.9% 883 of 29,967 level
ImagesiftBot ImageSift (Hive) training 2.7% 821 of 29,967 level
Timpibot Timpi training 2.7% 815 of 29,967 level
Scrapy Zyte (open-source framework) training 2.4% 713 of 29,967 level
AI2Bot Allen Institute for AI training 2.3% 698 of 29,967 level
cohere-training-data-crawler Cohere training 2.1% 620 of 29,967 level
OAI-SearchBot OpenAI answer index 2.0% 614 of 29,967 level
Meta-ExternalFetcher Meta live retrieval 1.9% 560 of 29,967 level
PanguBot Huawei training 1.7% 506 of 29,967 level
Claude-User Anthropic live retrieval 1.7% 503 of 29,967 level
DuckAssistBot DuckDuckGo live retrieval 1.7% 503 of 29,967 level
Perplexity-User Perplexity live retrieval 1.6% 483 of 29,967 level
Claude-SearchBot Anthropic answer index 1.6% 469 of 29,967 level
MistralAI-User Mistral AI live retrieval 1.5% 463 of 29,967 level
Kangaroo Bot Kangaroo LLM training 1.5% 439 of 29,967 level
Meta-WebIndexer Meta answer index 1.3% 404 of 29,967 level
Applebot Apple answer index 1.2% 352 of 29,967 level
Amzn-SearchBot Amazon answer index 0.9% 255 of 29,967 level
TikTokSpider ByteDance live retrieval 0.8% 247 of 29,967 level
Amzn-User Amazon live retrieval 0.7% 208 of 29,967 level
MistralAI-Index Mistral AI answer index 0.4% 112 of 29,967 level
facebookexternalhit Meta live retrieval 0.2% 57 of 29,967 level
Googlebot-News Google answer index 0.1% 39 of 29,967 level
Meta-ExternalAds Meta training 0.1% 39 of 29,967 level
Bingbot Microsoft answer index 0.1% 35 of 29,967 level
msnbot Microsoft answer index 0.1% 19 of 29,967 level
Googlebot Google answer index 0.1% 17 of 29,967 level
OAI-AdsBot OpenAI live retrieval 0.0% 14 of 29,967 level

Trend share of measured sites

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Where the blocking is share blocking an answer engine, by top-level domain

bar chart.uk (n=429)17.5%.fr (n=320)16.6%.de (n=823)16.5%.com (n=14603)9.8%.org (n=1840)9.2%.jp (n=401)9.2%.net (n=1306)9%.br (n=360)7.5%.in (n=302)5.6%.io (n=654)3.8%.edu (n=471)3%.ru (n=1157)2.9%

Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.

Changes recorded this week 200 total

Full change log · RSS

Cite this CC BY 4.0

The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.

Crawl Census, "The state of AI crawling, 2026-W45", 2026-11-08. https://crawlcensus.com/report/2026-W45

Machine-readable: census totals, full corpus, crawler registry. Method: methodology.