Weekly report · 2026-W42

The state of AI crawling

Measured on 29,997 reachable domains for the week ending 2026-10-18. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.

← 2026-W41  ·  2026-W43 →

29,997
domains measured
10%
block an answer engine
19%
refuse AI agents at the edge
15%
publish an llms.txt

Block rate by crawler with change on the previous week

CrawlerOperatorContent used forBlock rateSitesWeek on week
GPTBot OpenAI training 7.1% 2,169 of 30,399 level
CCBot Common Crawl Foundation archive 6.8% 2,062 of 30,399 level
Bytespider ByteDance training 6.5% 1,981 of 30,399 level
ClaudeBot Anthropic training 5.8% 1,767 of 30,399 level
Amazonbot Amazon training 4.8% 1,461 of 30,399 +0.1 pt
Applebot-Extended Apple training 4.8% 1,447 of 30,399 +0.2 pt
Meta-ExternalAgent Meta training 4.8% 1,446 of 30,399 level
Google-Extended Google training 4.7% 1,435 of 30,399 +0.1 pt
anthropic-ai Anthropic training 4.4% 1,346 of 30,399 level
omgilibot Webz.io training 4.4% 1,342 of 30,399 +0.1 pt
Diffbot Diffbot training 4.2% 1,272 of 30,399 +0.1 pt
cohere-ai Cohere live retrieval 4.1% 1,259 of 30,399 +0.1 pt
omgili Webz.io training 4.0% 1,230 of 30,399 +0.1 pt
SemrushBot-OCOB Semrush training 3.7% 1,124 of 30,399 -0.1 pt
ChatGPT-User OpenAI live retrieval 3.4% 1,036 of 30,399 -0.1 pt
PerplexityBot Perplexity answer index 3.4% 1,034 of 30,399 -0.1 pt
FacebookBot Meta training 3.2% 964 of 30,399 -0.1 pt
YouBot You.com answer index 3.0% 898 of 30,399 -0.1 pt
ImagesiftBot ImageSift (Hive) training 2.8% 841 of 30,399 level
Timpibot Timpi training 2.7% 826 of 30,399 level
Scrapy Zyte (open-source framework) training 2.4% 727 of 30,399 level
AI2Bot Allen Institute for AI training 2.3% 702 of 30,399 level
OAI-SearchBot OpenAI answer index 2.1% 629 of 30,399 -0.1 pt
cohere-training-data-crawler Cohere training 2.0% 617 of 30,399 level
Meta-ExternalFetcher Meta live retrieval 1.7% 530 of 30,399 level
DuckAssistBot DuckDuckGo live retrieval 1.7% 525 of 30,399 level
PanguBot Huawei training 1.7% 518 of 30,399 level
Claude-User Anthropic live retrieval 1.6% 498 of 30,399 level
Claude-SearchBot Anthropic answer index 1.6% 477 of 30,399 level
Google-CloudVertexBot Google live retrieval 1.6% 476 of 30,399 +0.1 pt
Perplexity-User Perplexity live retrieval 1.6% 474 of 30,399 level
MistralAI-User Mistral AI live retrieval 1.5% 465 of 30,399 level
Kangaroo Bot Kangaroo LLM training 1.5% 452 of 30,399 level
GoogleOther Google training 1.4% 434 of 30,399 +0.1 pt
Meta-WebIndexer Meta answer index 1.2% 361 of 30,399 level
Applebot Apple answer index 1.2% 353 of 30,399 level
TikTokSpider ByteDance live retrieval 0.9% 260 of 30,399 level
Amzn-SearchBot Amazon answer index 0.8% 242 of 30,399 level
MistralAI-Training Mistral AI training 0.7% 205 of 30,399 +0.1 pt
Amzn-User Amazon live retrieval 0.6% 182 of 30,399 level
MistralAI-Index Mistral AI answer index 0.3% 93 of 30,399 level
facebookexternalhit Meta live retrieval 0.2% 51 of 30,399 level
Bingbot Microsoft answer index 0.1% 38 of 30,399 level
Googlebot-News Google answer index 0.1% 36 of 30,399 level
Meta-ExternalAds Meta training 0.1% 24 of 30,399 level
msnbot Microsoft answer index 0.1% 19 of 30,399 level
Googlebot Google answer index 0.0% 15 of 30,399 level
OAI-AdsBot OpenAI live retrieval 0.0% 12 of 30,399 level

Trend share of measured sites

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Where the blocking is share blocking an answer engine, by top-level domain

bar chart.uk (n=429)17.5%.fr (n=320)16.6%.de (n=823)16.5%.com (n=14603)9.8%.org (n=1840)9.2%.jp (n=401)9.2%.net (n=1306)9%.br (n=360)7.5%.in (n=302)5.6%.io (n=654)3.8%.edu (n=471)3%.ru (n=1157)2.9%

Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.

Changes recorded this week 200 total

  • pik.ru Published an llms.txt file
  • biorxiv.org PerplexityBot is now refused at the network edge, whatever robots.txt permits
  • biorxiv.org AI access score moved 48 to 58
  • ubs.com AI access score moved 62 to 96
  • lixiang.com AI access score moved 56 to 46
  • ualberta.ca AI access score moved 77 to 47
  • defense.gov GPTBot is now refused at the network edge, whatever robots.txt permits
  • defense.gov OAI-SearchBot is now refused at the network edge, whatever robots.txt permits
  • defense.gov ClaudeBot is now refused at the network edge, whatever robots.txt permits
  • defense.gov PerplexityBot is now refused at the network edge, whatever robots.txt permits
  • defense.gov AI access score moved 54 to 67
  • bcg.com GPTBot is now refused at the network edge, whatever robots.txt permits
  • bcg.com OAI-SearchBot is now refused at the network edge, whatever robots.txt permits
  • bcg.com ClaudeBot is now refused at the network edge, whatever robots.txt permits
  • bcg.com PerplexityBot is now refused at the network edge, whatever robots.txt permits
  • bcg.com AI access score moved 58 to 75
  • lesechos.fr GPTBot is now blocked by robots.txt
  • lesechos.fr GPTBot is now refused at the network edge, whatever robots.txt permits
  • lesechos.fr OAI-SearchBot is now blocked by robots.txt
  • lesechos.fr OAI-SearchBot is now refused at the network edge, whatever robots.txt permits
  • lesechos.fr ChatGPT-User is now blocked by robots.txt
  • lesechos.fr ClaudeBot is now blocked by robots.txt
  • lesechos.fr ClaudeBot is now refused at the network edge, whatever robots.txt permits
  • lesechos.fr Claude-User is now blocked by robots.txt
  • lesechos.fr Claude-SearchBot is now blocked by robots.txt
  • lesechos.fr anthropic-ai is now blocked by robots.txt
  • lesechos.fr Google-Extended is now blocked by robots.txt
  • lesechos.fr Applebot is now blocked by robots.txt
  • lesechos.fr Applebot-Extended is now blocked by robots.txt
  • lesechos.fr PerplexityBot is now blocked by robots.txt

Full change log · RSS

Cite this CC BY 4.0

The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.

Crawl Census, "The state of AI crawling, 2026-W42", 2026-10-18. https://crawlcensus.com/report/2026-W42

Machine-readable: census totals, full corpus, crawler registry. Method: methodology.