Top-level domain segment

AI crawler access on .uk domains

95 of the 429 domains measured in this segment refuse a request from an identified AI user agent at the edge, 22% of the segment, against 17% whose robots.txt disallows an answer engine at the site root. The block that costs these sites citations is not the one written in the file. That answer-engine rate is 1.8 times the corpus rate of 10%, which is itself measured across 29,996 domains.

Mean AI access score across those 429 domains is 74.9 of 100, 44 of them publish an llms.txt (10%), and 233 publish structured data an answer engine can parse (54%).

429
domains measured
74.9
mean AI access score of 429
17%
block an answer engine (75 of 429)
10%
publish llms.txt (44 of 429)

Block rate by crawler, inside this segment robots.txt at the site root

CrawlerOperatorContent used forBlock rateBlockingMeasured
CCBot Common Crawl Foundation archive 17.2% 74 429
GPTBot OpenAI training 16.8% 72 429
ClaudeBot Anthropic training 16.6% 71 429
Applebot-Extended Apple training 16.1% 69 429
Meta-ExternalAgent Meta training 15.9% 68 429
anthropic-ai Anthropic training 14.5% 62 429
Bytespider ByteDance training 13.3% 57 429
YouBot You.com answer index 12.6% 54 429
PerplexityBot Perplexity answer index 11.7% 50 429
Amazonbot Amazon training 11.7% 50 429
Diffbot Diffbot training 10.7% 46 429
cohere-ai Cohere live retrieval 10.5% 45 429
OAI-SearchBot OpenAI answer index 10.3% 44 429
omgilibot Webz.io training 9.8% 42 429
omgili Webz.io training 9.3% 40 429
Google-Extended Google training 9.3% 40 429
Claude-SearchBot Anthropic answer index 6.8% 29 429
Timpibot Timpi training 6.5% 28 429
ImagesiftBot ImageSift (Hive) training 6.5% 28 429
ChatGPT-User OpenAI live retrieval 6.5% 28 429
MistralAI-User Mistral AI live retrieval 6.3% 27 429
Google-CloudVertexBot Google live retrieval 6.3% 27 429
DuckAssistBot DuckDuckGo live retrieval 6.3% 27 429
Claude-User Anthropic live retrieval 6.3% 27 429
Meta-ExternalFetcher Meta live retrieval 6.1% 26 429
Scrapy Zyte (open-source framework) training 5.8% 25 429
GoogleOther Google training 5.6% 24 429
SemrushBot-OCOB Semrush training 5.4% 23 429
PanguBot Huawei training 5.4% 23 429
cohere-training-data-crawler Cohere training 5.4% 23 429
AI2Bot Allen Institute for AI training 5.4% 23 429
Kangaroo Bot Kangaroo LLM training 4.7% 20 429
Applebot Apple answer index 4.4% 19 429
Perplexity-User Perplexity live retrieval 4.2% 18 429
MistralAI-Training Mistral AI training 3.7% 16 429
FacebookBot Meta training 3.5% 15 429
TikTokSpider ByteDance live retrieval 3.3% 14 429
Meta-WebIndexer Meta answer index 3.3% 14 429
MistralAI-Index Mistral AI answer index 1.2% 5 429
Amzn-User Amazon live retrieval 1.2% 5 429
Amzn-SearchBot Amazon answer index 1.2% 5 429
Meta-ExternalAds Meta training 0.2% 1 429
OAI-AdsBot OpenAI live retrieval 0.0% 0 429
msnbot Microsoft answer index 0.0% 0 429
Googlebot-News Google answer index 0.0% 0 429
Googlebot Google answer index 0.0% 0 429
facebookexternalhit Meta live retrieval 0.0% 0 429
Bingbot Microsoft answer index 0.0% 0 429

Block rate is the share of this segment's domains holding a policy record for that agent whose robots.txt disallows it at the site root, shown with its numerator and denominator. Denominators differ between agents because a record only exists once a domain has been measured for it. A domain serving no robots.txt counts as allowing everything, which is what the standard specifies.

Most accessible

DomainScoreEngines blockedllms.txt
1 newreg.co.uk 97 0 yes
2 reed.co.uk 97 0 yes
3 port.ac.uk 97 0 yes
4 postcodeanywhere.co.uk 97 0 yes
5 wellcome.ac.uk 96 0 yes
6 phrases.org.uk 95 0 no
7 lrb.co.uk 94 1 no
8 ebi.ac.uk 94 0 no
9 prat.uk 93 0 no
10 ageuk.org.uk 93 0 no
11 fasthosts.co.uk 93 0 yes
12 carwow.co.uk 93 0 no
13 smartsurvey.co.uk 93 0 yes
14 screamingfrog.co.uk 93 0 no
15 office.co.uk 93 0 yes

Highest AI access scores among the 429 domains measured in this segment.

Least accessible

DomainScoreEngines blockedllms.txt
1 twinkl.co.uk 37 4 no
2 49s.co.uk 39 2 no
3 amazon.co.uk 40 13 no
4 mailplus.co.uk 44 14 no
5 alaraby.co.uk 44 0 no
6 tvlicensing.co.uk 44 0 no
7 cargurus.co.uk 45 0 no
8 pinayflix.uk 46 15 no
9 coral.co.uk 46 0 no
10 etherealvpn.uk 46 0 no
11 weatheronline.co.uk 47 8 no
12 volnalink.uk 47 0 no
13 mirror.co.uk 47 3 no
14 manchestereveningnews.co.uk 47 3 no
15 walesonline.co.uk 47 3 no

Lowest AI access scores among the same 429 domains.

Compare with the rest of the grouping top-level domain segments

The largest 12 other segments in this grouping, ordered by size. The share beside each is that segment's own answer-engine block rate over its own count. Full table on the segments index.

What this segment does not tell you read this before citing it

A top-level domain is a weak proxy for jurisdiction. Most of them are open to any registrant anywhere, so a domain under a country-code registry may be operated from outside that country by a company no local law reaches. Read these rows as differences between registration communities and the publishing cultures inside them, not as differences between legal regimes.

Measure your own domain

The audit that produced every number on this page runs on any domain in about six seconds: robots.txt resolved for 48 agents, then live requests sent as four of them to see whether the edge agrees with the file.