Publishing platform segment

AI crawler access on hugo sites

6 of the 106 sites measured in this segment refuse a request from an identified AI user agent at the edge, 6% of the segment, against 4% whose robots.txt disallows an answer engine at the site root. The block that costs these sites citations is not the one written in the file. That answer-engine rate is 5.8 points below the corpus rate of 10%, which is itself measured across 29,997 domains.

Mean AI access score across those 106 sites is 76.9 of 100, 13 of them publish an llms.txt (12%), and 35 publish structured data an answer engine can parse (33%).

106
sites measured
76.9
mean AI access score of 106
4%
block an answer engine (4 of 106)
12%
publish llms.txt (13 of 106)

Block rate by crawler, inside this segment robots.txt at the site root

CrawlerOperatorContent used forBlock rateBlockingMeasured
GPTBot OpenAI training 3.8% 4 106
CCBot Common Crawl Foundation archive 3.8% 4 106
omgilibot Webz.io training 2.8% 3 106
omgili Webz.io training 2.8% 3 106
MistralAI-Training Mistral AI training 2.8% 3 106
Meta-ExternalAgent Meta training 2.8% 3 106
GoogleOther Google training 2.8% 3 106
Google-Extended Google training 2.8% 3 106
Google-CloudVertexBot Google live retrieval 2.8% 3 106
Diffbot Diffbot training 2.8% 3 106
cohere-ai Cohere live retrieval 2.8% 3 106
ClaudeBot Anthropic training 2.8% 3 106
ChatGPT-User OpenAI live retrieval 2.8% 3 106
Bytespider ByteDance training 2.8% 3 106
Applebot-Extended Apple training 2.8% 3 106
anthropic-ai Anthropic training 2.8% 3 106
Amazonbot Amazon training 2.8% 3 106
YouBot You.com answer index 1.9% 2 106
PerplexityBot Perplexity answer index 1.9% 2 106
Perplexity-User Perplexity live retrieval 1.9% 2 106
OAI-SearchBot OpenAI answer index 1.9% 2 106
MistralAI-User Mistral AI live retrieval 1.9% 2 106
MistralAI-Index Mistral AI answer index 1.9% 2 106
Meta-ExternalFetcher Meta live retrieval 1.9% 2 106
Claude-User Anthropic live retrieval 1.9% 2 106
Claude-SearchBot Anthropic answer index 1.9% 2 106
Applebot Apple answer index 1.9% 2 106
Amzn-User Amazon live retrieval 1.9% 2 106
Amzn-SearchBot Amazon answer index 1.9% 2 106
SemrushBot-OCOB Semrush training 0.9% 1 106
Timpibot Timpi training 0.0% 0 106
TikTokSpider ByteDance live retrieval 0.0% 0 106
Scrapy Zyte (open-source framework) training 0.0% 0 106
PanguBot Huawei training 0.0% 0 106
OAI-AdsBot OpenAI live retrieval 0.0% 0 106
msnbot Microsoft answer index 0.0% 0 106
Meta-WebIndexer Meta answer index 0.0% 0 106
Meta-ExternalAds Meta training 0.0% 0 106
Kangaroo Bot Kangaroo LLM training 0.0% 0 106
ImagesiftBot ImageSift (Hive) training 0.0% 0 106
Googlebot-News Google answer index 0.0% 0 106
Googlebot Google answer index 0.0% 0 106
facebookexternalhit Meta live retrieval 0.0% 0 106
FacebookBot Meta training 0.0% 0 106
DuckAssistBot DuckDuckGo live retrieval 0.0% 0 106
cohere-training-data-crawler Cohere training 0.0% 0 106
Bingbot Microsoft answer index 0.0% 0 106
AI2Bot Allen Institute for AI training 0.0% 0 106

Block rate is the share of this segment's domains holding a policy record for that agent whose robots.txt disallows it at the site root, shown with its numerator and denominator. Denominators differ between agents because a record only exists once a domain has been measured for it. A domain serving no robots.txt counts as allowing everything, which is what the standard specifies.

Most accessible

DomainScoreEngines blockedllms.txt
1 netdata.cloud 99 0 yes
2 packetstream.io 99 0 yes
3 postaffiliatepro.com 98 0 yes
4 hosting.de 97 0 yes
5 ladesk.com 97 0 yes
6 cumulocity.com 97 0 yes
7 ipbase.com 97 0 yes
8 carto.com 97 0 yes
9 selenium.dev 97 0 yes
10 privadovpn.com 95 0 yes
11 privado.io 95 0 yes
12 brave.com 95 0 no
13 rackcorp.com 95 0 no
14 centerdevice.de 93 0 no
15 modsecurity.org 93 0 no

Highest AI access scores among the 106 sites measured in this segment.

Least accessible

DomainScoreEngines blockedllms.txt
1 notepad-plus-plus.org 51 0 no
2 wvu.edu 56 0 no
3 mauribac.com 58 0 no
4 attendanceondemand.com 60 2 no
5 git-scm.com 60 0 no
6 scipy.org 61 0 no
7 libertad.link 61 0 no
8 ethnologue.com 63 0 no
9 memcached.org 63 0 no
10 int08h.com 64 0 no
11 barackobama.com 64 0 no
12 dweb.link 65 15 no
13 ipfs.io 65 15 no
14 syncthing.net 66 0 no
15 royalarkgames.com 66 0 no

Lowest AI access scores among the same 106 sites.

Compare with the rest of the grouping publishing platform segments

The largest 10 other segments in this grouping, ordered by size. The share beside each is that segment's own answer-engine block rate over its own count. Full table on the segments index.

What this segment does not tell you read this before citing it

Platform detection relies on a generator meta tag, so coverage is partial and skewed. A site that strips the tag, or a system that never writes one, is invisible here. These rows therefore over-represent platforms that advertise themselves and say nothing at all about the ones that do not.

Measure your own domain

The audit that produced every number on this page runs on any domain in about six seconds: robots.txt resolved for 48 agents, then live requests sent as four of them to see whether the edge agrees with the file.