Weekly report · 2026-W46

The state of AI crawling

Measured on 29,997 reachable domains for the week ending 2026-11-15. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.

← 2026-W45  ·  2026-W47 →

29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt

Block rate by crawler with change on the previous week

CrawlerOperatorContent used forBlock rateSitesWeek on week
GPTBot OpenAI training 9.8% 2,937 of 29,982 level
CCBot Common Crawl Foundation archive 9.6% 2,878 of 29,982 level
Bytespider ByteDance training 9.4% 2,812 of 29,982 level
ClaudeBot Anthropic training 8.6% 2,574 of 29,982 level
Meta-ExternalAgent Meta training 7.8% 2,340 of 29,982 level
Amazonbot Amazon training 7.6% 2,290 of 29,982 level
Applebot-Extended Apple training 7.6% 2,288 of 29,982 level
Google-Extended Google training 7.6% 2,268 of 29,982 level
anthropic-ai Anthropic training 7.3% 2,190 of 29,982 level
omgilibot Webz.io training 7.2% 2,166 of 29,982 level
Diffbot Diffbot training 7.1% 2,121 of 29,982 level
cohere-ai Cohere live retrieval 7.1% 2,114 of 29,982 level
omgili Webz.io training 6.9% 2,057 of 29,982 level
Google-CloudVertexBot Google live retrieval 4.5% 1,360 of 29,982 level
GoogleOther Google training 4.4% 1,311 of 29,982 level
MistralAI-Training Mistral AI training 3.7% 1,124 of 29,982 level
SemrushBot-OCOB Semrush training 3.5% 1,058 of 29,982 level
PerplexityBot Perplexity answer index 3.4% 1,014 of 29,982 level
ChatGPT-User OpenAI live retrieval 3.3% 991 of 29,982 level
FacebookBot Meta training 3.1% 923 of 29,982 level
YouBot You.com answer index 2.9% 882 of 29,982 level
ImagesiftBot ImageSift (Hive) training 2.7% 818 of 29,982 level
Timpibot Timpi training 2.7% 814 of 29,982 level
Scrapy Zyte (open-source framework) training 2.4% 711 of 29,982 level
AI2Bot Allen Institute for AI training 2.3% 697 of 29,982 level
cohere-training-data-crawler Cohere training 2.1% 619 of 29,982 level
OAI-SearchBot OpenAI answer index 2.1% 615 of 29,982 level
Meta-ExternalFetcher Meta live retrieval 1.9% 560 of 29,982 level
PanguBot Huawei training 1.7% 506 of 29,982 level
Claude-User Anthropic live retrieval 1.7% 505 of 29,982 level
DuckAssistBot DuckDuckGo live retrieval 1.7% 504 of 29,982 level
Perplexity-User Perplexity live retrieval 1.6% 483 of 29,982 level
Claude-SearchBot Anthropic answer index 1.6% 469 of 29,982 level
MistralAI-User Mistral AI live retrieval 1.5% 464 of 29,982 level
Kangaroo Bot Kangaroo LLM training 1.5% 437 of 29,982 level
Meta-WebIndexer Meta answer index 1.4% 407 of 29,982 level
Applebot Apple answer index 1.2% 351 of 29,982 level
Amzn-SearchBot Amazon answer index 0.9% 255 of 29,982 level
TikTokSpider ByteDance live retrieval 0.8% 246 of 29,982 level
Amzn-User Amazon live retrieval 0.7% 209 of 29,982 level
MistralAI-Index Mistral AI answer index 0.4% 112 of 29,982 level
facebookexternalhit Meta live retrieval 0.2% 56 of 29,982 level
Googlebot-News Google answer index 0.1% 39 of 29,982 level
Meta-ExternalAds Meta training 0.1% 39 of 29,982 level
Bingbot Microsoft answer index 0.1% 35 of 29,982 level
msnbot Microsoft answer index 0.1% 18 of 29,982 level
Googlebot Google answer index 0.1% 17 of 29,982 level
OAI-AdsBot OpenAI live retrieval 0.0% 14 of 29,982 level

Trend share of measured sites

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Where the blocking is share blocking an answer engine, by top-level domain

bar chart.uk (n=429)17.5%.fr (n=320)16.6%.de (n=823)16.5%.com (n=14603)9.8%.org (n=1840)9.2%.jp (n=401)9.2%.net (n=1306)9%.br (n=360)7.5%.in (n=302)5.6%.io (n=654)3.8%.edu (n=471)3%.ru (n=1157)2.9%

Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.

Changes recorded this week 200 total

  • robinhood.com Published an llms.txt file
  • kir2kos.net Published an llms.txt file
  • nexon.com AI access score moved 59 to 49
  • bgtee.com GPTBot is now blocked by robots.txt
  • bgtee.com ClaudeBot is now blocked by robots.txt
  • bgtee.com anthropic-ai is now blocked by robots.txt
  • bgtee.com Google-Extended is now blocked by robots.txt
  • bgtee.com Google-CloudVertexBot is now blocked by robots.txt
  • bgtee.com GoogleOther is now blocked by robots.txt
  • bgtee.com Applebot-Extended is now blocked by robots.txt
  • bgtee.com PerplexityBot is now refused at the network edge, whatever robots.txt permits
  • bgtee.com Meta-ExternalAgent is now blocked by robots.txt
  • bgtee.com Bytespider is now blocked by robots.txt
  • bgtee.com Amazonbot is now blocked by robots.txt
  • bgtee.com CCBot is now blocked by robots.txt
  • bgtee.com Diffbot is now blocked by robots.txt
  • bgtee.com omgili is now blocked by robots.txt
  • bgtee.com omgilibot is now blocked by robots.txt
  • bgtee.com cohere-ai is now blocked by robots.txt
  • bgtee.com MistralAI-Training is now blocked by robots.txt
  • mamastar.jp AI access score moved 74 to 49
  • launchpad.net OAI-SearchBot is no longer refused at the network edge
  • launchpad.net ClaudeBot is no longer refused at the network edge
  • launchpad.net PerplexityBot is no longer refused at the network edge
  • launchpad.net AI access score moved 39 to 52
  • iea.org AI access score moved 73 to 88
  • imageshack.us MistralAI-Training is now blocked by robots.txt
  • tn.gov.in AI access score moved 65 to 46
  • rajasthan.gov.in Published an llms.txt file
  • square.site ClaudeBot is now refused at the network edge, whatever robots.txt permits

Full change log · RSS

Cite this CC BY 4.0

The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.

Crawl Census, "The state of AI crawling, 2026-W46", 2026-11-15. https://crawlcensus.com/report/2026-W46

Machine-readable: census totals, full corpus, crawler registry. Method: methodology.