Weekly report · 2026-W22

The state of AI crawling

Measured on 29,998 reachable domains for the week ending 2026-05-31. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.

← 2026-W21  ·  2026-W23 →

29,998
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt

Block rate by crawler with change on the previous week

CrawlerOperatorContent used forBlock rateSitesWeek on week
GPTBot OpenAI training 9.8% 2,937 of 29,991 new
CCBot Common Crawl Foundation archive 9.6% 2,881 of 29,991 new
Bytespider ByteDance training 9.4% 2,817 of 29,991 new
ClaudeBot Anthropic training 8.6% 2,574 of 29,991 new
Meta-ExternalAgent Meta training 7.8% 2,340 of 29,991 new
Amazonbot Amazon training 7.7% 2,298 of 29,991 new
Applebot-Extended Apple training 7.6% 2,291 of 29,991 new
Google-Extended Google training 7.6% 2,269 of 29,991 new
anthropic-ai Anthropic training 7.3% 2,193 of 29,991 new
omgilibot Webz.io training 7.2% 2,164 of 29,991 new
Diffbot Diffbot training 7.1% 2,120 of 29,991 new
cohere-ai Cohere live retrieval 7.0% 2,112 of 29,991 new
omgili Webz.io training 6.9% 2,055 of 29,991 new
Google-CloudVertexBot Google live retrieval 4.5% 1,361 of 29,991 new
GoogleOther Google training 4.4% 1,314 of 29,991 new
MistralAI-Training Mistral AI training 3.8% 1,130 of 29,991 new
SemrushBot-OCOB Semrush training 3.5% 1,059 of 29,991 new
PerplexityBot Perplexity answer index 3.4% 1,006 of 29,991 new
ChatGPT-User OpenAI live retrieval 3.3% 983 of 29,991 new
FacebookBot Meta training 3.1% 919 of 29,991 new
YouBot You.com answer index 2.9% 876 of 29,991 new
ImagesiftBot ImageSift (Hive) training 2.7% 817 of 29,991 new
Timpibot Timpi training 2.7% 811 of 29,991 new
Scrapy Zyte (open-source framework) training 2.4% 706 of 29,991 new
AI2Bot Allen Institute for AI training 2.3% 692 of 29,991 new
cohere-training-data-crawler Cohere training 2.1% 615 of 29,991 new
OAI-SearchBot OpenAI answer index 2.0% 608 of 29,991 new
Meta-ExternalFetcher Meta live retrieval 1.8% 553 of 29,991 new
PanguBot Huawei training 1.7% 502 of 29,991 new
Claude-User Anthropic live retrieval 1.7% 498 of 29,991 new
DuckAssistBot DuckDuckGo live retrieval 1.7% 498 of 29,991 new
Perplexity-User Perplexity live retrieval 1.6% 476 of 29,991 new
Claude-SearchBot Anthropic answer index 1.5% 463 of 29,991 new
MistralAI-User Mistral AI live retrieval 1.5% 457 of 29,991 new
Kangaroo Bot Kangaroo LLM training 1.4% 433 of 29,991 new
Meta-WebIndexer Meta answer index 1.3% 402 of 29,991 new
Applebot Apple answer index 1.2% 350 of 29,991 new
Amzn-SearchBot Amazon answer index 0.8% 253 of 29,991 new
TikTokSpider ByteDance live retrieval 0.8% 247 of 29,991 new
Amzn-User Amazon live retrieval 0.7% 208 of 29,991 new
MistralAI-Index Mistral AI answer index 0.4% 111 of 29,991 new
facebookexternalhit Meta live retrieval 0.2% 56 of 29,991 new
Meta-ExternalAds Meta training 0.1% 41 of 29,991 new
Googlebot-News Google answer index 0.1% 39 of 29,991 new
Bingbot Microsoft answer index 0.1% 35 of 29,991 new
msnbot Microsoft answer index 0.1% 18 of 29,991 new
Googlebot Google answer index 0.1% 17 of 29,991 new
OAI-AdsBot OpenAI live retrieval 0.0% 13 of 29,991 new

Trend share of measured sites

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Where the blocking is share blocking an answer engine, by top-level domain

bar chart.uk (n=429)17.5%.fr (n=320)16.6%.de (n=823)16.5%.com (n=14604)9.8%.org (n=1840)9.2%.jp (n=401)9.2%.net (n=1306)9%.br (n=360)7.5%.in (n=302)5.6%.io (n=654)3.8%.edu (n=471)3%.ru (n=1157)2.9%

Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.

Changes recorded this week 0 total

No crawler policy changes were recorded in this week.

Full change log · RSS

Cite this CC BY 4.0

The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.

Crawl Census, "The state of AI crawling, 2026-W22", 2026-05-31. https://crawlcensus.com/report/2026-W22

Machine-readable: census totals, full corpus, crawler registry. Method: methodology.