Edge provider segment

AI crawler access behind Azure Front Door

27 of the 293 origins measured in this segment refuse a request from an identified AI user agent at the edge, 9% of the segment, against 7% whose robots.txt disallows an answer engine at the site root. The block that costs these sites citations is not the one written in the file. That answer-engine rate is 2.7 points below the corpus rate of 10%, which is itself measured across 29,997 domains.

Mean AI access score across those 293 origins is 75.2 of 100, 30 of them publish an llms.txt (10%), and 125 publish structured data an answer engine can parse (43%).

293
origins measured
75.2
mean AI access score of 293
7%
block an answer engine (20 of 293)
10%
publish llms.txt (30 of 293)

Block rate by crawler, inside this segment robots.txt at the site root

CrawlerOperatorContent used forBlock rateBlockingMeasured
GPTBot OpenAI training 8.2% 24 293
Bytespider ByteDance training 8.2% 24 293
CCBot Common Crawl Foundation archive 7.2% 21 293
Meta-ExternalAgent Meta training 6.8% 20 293
ClaudeBot Anthropic training 6.5% 19 293
SemrushBot-OCOB Semrush training 6.1% 18 293
Applebot-Extended Apple training 6.1% 18 293
Amazonbot Amazon training 6.1% 18 293
cohere-ai Cohere live retrieval 5.8% 17 293
Scrapy Zyte (open-source framework) training 5.5% 16 293
Diffbot Diffbot training 5.5% 16 293
anthropic-ai Anthropic training 5.5% 16 293
YouBot You.com answer index 5.1% 15 293
Timpibot Timpi training 5.1% 15 293
omgilibot Webz.io training 5.1% 15 293
PerplexityBot Perplexity answer index 4.8% 14 293
omgili Webz.io training 4.8% 14 293
MistralAI-User Mistral AI live retrieval 4.8% 14 293
ImagesiftBot ImageSift (Hive) training 4.8% 14 293
Claude-SearchBot Anthropic answer index 4.8% 14 293
OAI-SearchBot OpenAI answer index 4.4% 13 293
ChatGPT-User OpenAI live retrieval 4.4% 13 293
TikTokSpider ByteDance live retrieval 4.1% 12 293
PanguBot Huawei training 4.1% 12 293
Meta-ExternalFetcher Meta live retrieval 4.1% 12 293
Kangaroo Bot Kangaroo LLM training 4.1% 12 293
DuckAssistBot DuckDuckGo live retrieval 4.1% 12 293
cohere-training-data-crawler Cohere training 4.1% 12 293
Claude-User Anthropic live retrieval 4.1% 12 293
AI2Bot Allen Institute for AI training 4.1% 12 293
Applebot Apple answer index 3.8% 11 293
Google-Extended Google training 2.4% 7 293
FacebookBot Meta training 1.7% 5 293
Perplexity-User Perplexity live retrieval 0.7% 2 293
GoogleOther Google training 0.7% 2 293
Google-CloudVertexBot Google live retrieval 0.7% 2 293
MistralAI-Training Mistral AI training 0.3% 1 293
Googlebot-News Google answer index 0.3% 1 293
Bingbot Microsoft answer index 0.3% 1 293
OAI-AdsBot OpenAI live retrieval 0.0% 0 293
msnbot Microsoft answer index 0.0% 0 293
MistralAI-Index Mistral AI answer index 0.0% 0 293
Meta-WebIndexer Meta answer index 0.0% 0 293
Meta-ExternalAds Meta training 0.0% 0 293
Googlebot Google answer index 0.0% 0 293
facebookexternalhit Meta live retrieval 0.0% 0 293
Amzn-User Amazon live retrieval 0.0% 0 293
Amzn-SearchBot Amazon answer index 0.0% 0 293

Block rate is the share of this segment's domains holding a policy record for that agent whose robots.txt disallows it at the site root, shown with its numerator and denominator. Denominators differ between agents because a record only exists once a domain has been measured for it. A domain serving no robots.txt counts as allowing everything, which is what the standard specifies.

Most accessible

DomainScoreEngines blockedllms.txt
1 arvato-systems.de 98 0 yes
2 zenoti.com 97 0 yes
3 myregistry.com 96 0 yes
4 dormakaba.com 95 0 yes
5 cirquedusoleil.com 95 0 yes
6 arkadium.com 95 0 yes
7 tobii.com 94 0 yes
8 natwest.com 93 0 yes
9 pampers.com 93 0 yes
10 lexware.de 93 0 yes
11 affordablehousing.com 92 0 no
12 maif.fr 92 0 no
13 teleticket.com.pe 92 0 no
14 frontiersin.org 92 0 yes
15 piraeusbank.gr 91 0 no

Highest AI access scores among the 293 origins measured in this segment.

Least accessible

DomainScoreEngines blockedllms.txt
1 github.dev 46 0 no
2 switch.ch 46 0 no
3 comed.com 47 0 no
4 clipro.tv 47 0 no
5 skynet.be 47 0 no
6 aph.gov.au 49 0 no
7 whiteboard.ms 49 0 no
8 wbd.ms 49 0 no
9 tvhot2.com 50 0 no
10 noradsanta.org 50 0 no
11 powerautomate.com 50 0 no
12 laccd.edu 52 0 no
13 umbraco.io 52 0 no
14 openlearning.com 52 0 no
15 microsoftcasualgames.com 52 0 no

Lowest AI access scores among the same 293 origins.

Compare with the rest of the grouping edge provider segments

The largest 10 other segments in this grouping, ordered by size. The share beside each is that segment's own answer-engine block rate over its own count. Full table on the segments index.

What this segment does not tell you read this before citing it

Only providers that expose an identifying response header can be detected. Sites behind an edge that stays quiet, or behind an origin that strips those headers, are absent from the count, so this sample is not representative of any provider's customer base.

Measure your own domain

The audit that produced every number on this page runs on any domain in about six seconds: robots.txt resolved for 48 agents, then live requests sent as four of them to see whether the edge agrees with the file.