Weekly report · 2026-W47 · current

The state of AI crawling

Measured on 29,997 reachable domains for the week ending 2026-11-22. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.

← 2026-W46  ·  this is the latest

29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt

Block rate by crawler with change on the previous week

CrawlerOperatorContent used forBlock rateSitesWeek on week
GPTBot OpenAI training 9.8% 2,937 of 29,991 level
CCBot Common Crawl Foundation archive 9.6% 2,881 of 29,991 level
Bytespider ByteDance training 9.4% 2,817 of 29,991 level
ClaudeBot Anthropic training 8.6% 2,574 of 29,991 level
Meta-ExternalAgent Meta training 7.8% 2,340 of 29,991 level
Amazonbot Amazon training 7.7% 2,298 of 29,991 level
Applebot-Extended Apple training 7.6% 2,291 of 29,991 level
Google-Extended Google training 7.6% 2,269 of 29,991 level
anthropic-ai Anthropic training 7.3% 2,193 of 29,991 level
omgilibot Webz.io training 7.2% 2,164 of 29,991 level
Diffbot Diffbot training 7.1% 2,120 of 29,991 level
cohere-ai Cohere live retrieval 7.0% 2,112 of 29,991 level
omgili Webz.io training 6.9% 2,055 of 29,991 level
Google-CloudVertexBot Google live retrieval 4.5% 1,361 of 29,991 level
GoogleOther Google training 4.4% 1,314 of 29,991 level
MistralAI-Training Mistral AI training 3.8% 1,130 of 29,991 level
SemrushBot-OCOB Semrush training 3.5% 1,059 of 29,991 level
PerplexityBot Perplexity answer index 3.4% 1,006 of 29,991 level
ChatGPT-User OpenAI live retrieval 3.3% 983 of 29,991 level
FacebookBot Meta training 3.1% 919 of 29,991 level
YouBot You.com answer index 2.9% 876 of 29,991 level
ImagesiftBot ImageSift (Hive) training 2.7% 817 of 29,991 level
Timpibot Timpi training 2.7% 811 of 29,991 level
Scrapy Zyte (open-source framework) training 2.4% 706 of 29,991 level
AI2Bot Allen Institute for AI training 2.3% 692 of 29,991 level
cohere-training-data-crawler Cohere training 2.1% 615 of 29,991 level
OAI-SearchBot OpenAI answer index 2.0% 608 of 29,991 level
Meta-ExternalFetcher Meta live retrieval 1.8% 553 of 29,991 level
PanguBot Huawei training 1.7% 502 of 29,991 level
Claude-User Anthropic live retrieval 1.7% 498 of 29,991 level
DuckAssistBot DuckDuckGo live retrieval 1.7% 498 of 29,991 level
Perplexity-User Perplexity live retrieval 1.6% 476 of 29,991 level
Claude-SearchBot Anthropic answer index 1.5% 463 of 29,991 level
MistralAI-User Mistral AI live retrieval 1.5% 457 of 29,991 level
Kangaroo Bot Kangaroo LLM training 1.4% 433 of 29,991 level
Meta-WebIndexer Meta answer index 1.3% 402 of 29,991 level
Applebot Apple answer index 1.2% 350 of 29,991 level
Amzn-SearchBot Amazon answer index 0.8% 253 of 29,991 level
TikTokSpider ByteDance live retrieval 0.8% 247 of 29,991 level
Amzn-User Amazon live retrieval 0.7% 208 of 29,991 level
MistralAI-Index Mistral AI answer index 0.4% 111 of 29,991 level
facebookexternalhit Meta live retrieval 0.2% 56 of 29,991 level
Meta-ExternalAds Meta training 0.1% 41 of 29,991 level
Googlebot-News Google answer index 0.1% 39 of 29,991 level
Bingbot Microsoft answer index 0.1% 35 of 29,991 level
msnbot Microsoft answer index 0.1% 18 of 29,991 level
Googlebot Google answer index 0.1% 17 of 29,991 level
OAI-AdsBot OpenAI live retrieval 0.0% 13 of 29,991 level

Trend share of measured sites

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Where the blocking is share blocking an answer engine, by top-level domain

bar chart.uk (n=429)17.5%.fr (n=320)16.6%.de (n=823)16.5%.com (n=14603)9.8%.org (n=1840)9.2%.jp (n=401)9.2%.net (n=1306)9%.br (n=360)7.5%.in (n=302)5.6%.io (n=654)3.8%.edu (n=471)3%.ru (n=1157)2.9%

Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.

Changes recorded this week 200 total

  • gm.com GPTBot is no longer refused at the network edge
  • gm.com OAI-SearchBot is no longer refused at the network edge
  • gm.com ClaudeBot is no longer refused at the network edge
  • gm.com PerplexityBot is no longer refused at the network edge
  • shopee.co.th ClaudeBot is now refused at the network edge, whatever robots.txt permits
  • amazon.sa GPTBot is now allowed by robots.txt
  • amazon.sa OAI-SearchBot is now allowed by robots.txt
  • amazon.sa ChatGPT-User is now allowed by robots.txt
  • amazon.sa ClaudeBot is now allowed by robots.txt
  • amazon.sa Claude-User is now allowed by robots.txt
  • amazon.sa Claude-SearchBot is now allowed by robots.txt
  • amazon.sa Google-Extended is now allowed by robots.txt
  • amazon.sa Google-CloudVertexBot is now allowed by robots.txt
  • amazon.sa GoogleOther is now allowed by robots.txt
  • amazon.sa PerplexityBot is now allowed by robots.txt
  • amazon.sa Perplexity-User is now allowed by robots.txt
  • amazon.sa Meta-ExternalAgent is now allowed by robots.txt
  • amazon.sa Meta-ExternalFetcher is now allowed by robots.txt
  • amazon.sa Meta-WebIndexer is now allowed by robots.txt
  • amazon.sa Bytespider is now allowed by robots.txt
  • amazon.sa CCBot is now allowed by robots.txt
  • amazon.sa Diffbot is now allowed by robots.txt
  • amazon.sa omgili is now allowed by robots.txt
  • amazon.sa omgilibot is now allowed by robots.txt
  • amazon.sa AI2Bot is now allowed by robots.txt
  • amazon.sa cohere-ai is now allowed by robots.txt
  • amazon.sa cohere-training-data-crawler is now allowed by robots.txt
  • amazon.sa MistralAI-User is now allowed by robots.txt
  • amazon.sa DuckAssistBot is now allowed by robots.txt
  • amazon.sa YouBot is now allowed by robots.txt

Full change log · RSS

Cite this CC BY 4.0

The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.

Crawl Census, "The state of AI crawling, 2026-W47", 2026-11-22. https://crawlcensus.com/report/2026-W47

Machine-readable: census totals, full corpus, crawler registry. Method: methodology.