Publishing platform segment

AI crawler access on drupal sites

179 of the 829 sites measured in this segment refuse a request from an identified AI user agent at the edge, 22% of the segment, against 7% whose robots.txt disallows an answer engine at the site root. The block that costs these sites citations is not the one written in the file. That answer-engine rate is 3.0 points below the corpus rate of 10%, which is itself measured across 29,997 domains.

Mean AI access score across those 829 sites is 75.7 of 100, 61 of them publish an llms.txt (7%), and 267 publish structured data an answer engine can parse (32%).

829
sites measured
75.7
mean AI access score of 829
7%
block an answer engine (54 of 829)
7%
publish llms.txt (61 of 829)

Block rate by crawler, inside this segment robots.txt at the site root

CrawlerOperatorContent used forBlock rateBlockingMeasured
GPTBot OpenAI training 6.4% 53 829
Bytespider ByteDance training 6.4% 53 829
ClaudeBot Anthropic training 5.8% 48 829
CCBot Common Crawl Foundation archive 5.4% 45 829
anthropic-ai Anthropic training 5.1% 42 829
Applebot-Extended Apple training 4.9% 41 829
Google-Extended Google training 4.6% 38 829
Diffbot Diffbot training 4.6% 38 829
cohere-ai Cohere live retrieval 4.6% 38 829
omgilibot Webz.io training 4.5% 37 829
omgili Webz.io training 4.5% 37 829
Amazonbot Amazon training 4.5% 37 829
Meta-ExternalAgent Meta training 4.3% 36 829
SemrushBot-OCOB Semrush training 3.6% 30 829
Google-CloudVertexBot Google live retrieval 3.0% 25 829
GoogleOther Google training 2.9% 24 829
MistralAI-Training Mistral AI training 2.5% 21 829
ChatGPT-User OpenAI live retrieval 2.5% 21 829
PerplexityBot Perplexity answer index 2.1% 17 829
ImagesiftBot ImageSift (Hive) training 1.8% 15 829
YouBot You.com answer index 1.6% 13 829
Scrapy Zyte (open-source framework) training 1.6% 13 829
FacebookBot Meta training 1.6% 13 829
Timpibot Timpi training 1.4% 12 829
OAI-SearchBot OpenAI answer index 1.3% 11 829
Applebot Apple answer index 1.1% 9 829
AI2Bot Allen Institute for AI training 1.1% 9 829
Meta-ExternalFetcher Meta live retrieval 1.0% 8 829
cohere-training-data-crawler Cohere training 0.8% 7 829
TikTokSpider ByteDance live retrieval 0.7% 6 829
MistralAI-User Mistral AI live retrieval 0.7% 6 829
Meta-WebIndexer Meta answer index 0.7% 6 829
Claude-User Anthropic live retrieval 0.7% 6 829
Perplexity-User Perplexity live retrieval 0.6% 5 829
PanguBot Huawei training 0.6% 5 829
Kangaroo Bot Kangaroo LLM training 0.6% 5 829
Amzn-User Amazon live retrieval 0.6% 5 829
Claude-SearchBot Anthropic answer index 0.5% 4 829
Amzn-SearchBot Amazon answer index 0.5% 4 829
facebookexternalhit Meta live retrieval 0.4% 3 829
DuckAssistBot DuckDuckGo live retrieval 0.4% 3 829
Bingbot Microsoft answer index 0.4% 3 829
OAI-AdsBot OpenAI live retrieval 0.1% 1 829
MistralAI-Index Mistral AI answer index 0.1% 1 829
msnbot Microsoft answer index 0.0% 0 829
Meta-ExternalAds Meta training 0.0% 0 829
Googlebot-News Google answer index 0.0% 0 829
Googlebot Google answer index 0.0% 0 829

Block rate is the share of this segment's domains holding a policy record for that agent whose robots.txt disallows it at the site root, shown with its numerator and denominator. Denominators differ between agents because a record only exists once a domain has been measured for it. A domain serving no robots.txt counts as allowing everything, which is what the standard specifies.

Most accessible

DomainScoreEngines blockedllms.txt
1 sahistory.org.za 99 0 yes
2 voya.com 98 0 yes
3 wri.org 98 0 yes
4 posit.co 97 0 yes
5 skynettechnologies.com 97 0 yes
6 port.ac.uk 97 0 yes
7 acquia.com 96 0 yes
8 monsido.com 96 0 yes
9 instructure.com 96 0 yes
10 belvedere.at 96 0 yes
11 flexera.com 95 0 no
12 mtholyoke.edu 95 0 no
13 ri.se 95 0 no
14 altium.com 95 0 yes
15 starwoodhotels.com 94 0 yes

Highest AI access scores among the 829 sites measured in this segment.

Least accessible

DomainScoreEngines blockedllms.txt
1 biorxiv.org 44 3 no
2 hhmi.org 45 0 no
3 icj-cij.org 47 0 no
4 taxpolicycenter.org 50 0 no
5 nic.aero 50 0 no
6 fraserinstitute.org 51 6 no
7 colorado.gov 51 0 no
8 laccd.edu 52 0 no
9 unmissions.org 52 0 no
10 anses.fr 53 0 no
11 saharareporters.com 53 2 no
12 fincen.gov 53 0 no
13 penguinrandomhouse.ca 53 0 no
14 naeyc.org 54 0 no
15 justice.gov 54 6 no

Lowest AI access scores among the same 829 sites.

Compare with the rest of the grouping publishing platform segments

The largest 10 other segments in this grouping, ordered by size. The share beside each is that segment's own answer-engine block rate over its own count. Full table on the segments index.

What this segment does not tell you read this before citing it

Platform detection relies on a generator meta tag, so coverage is partial and skewed. A site that strips the tag, or a system that never writes one, is invisible here. These rows therefore over-represent platforms that advertise themselves and say nothing at all about the ones that do not.

Measure your own domain

The audit that produced every number on this page runs on any domain in about six seconds: robots.txt resolved for 48 agents, then live requests sent as four of them to see whether the edge agrees with the file.