Top-level domain segment

AI crawler access on .com domains

2,926 of the 14,603 domains measured in this segment refuse a request from an identified AI user agent at the edge, 20% of the segment, against 10% whose robots.txt disallows an answer engine at the site root. The block that costs these sites citations is not the one written in the file. That answer-engine rate is level with the corpus rate of 10%, which is itself measured across 29,997 domains.

Mean AI access score across those 14,603 domains is 74.5 of 100, 2,605 of them publish an llms.txt (18%), and 7,816 publish structured data an answer engine can parse (54%).

14,603
domains measured
74.5
mean AI access score of 14,603
10%
block an answer engine (1,430 of 14,603)
18%
publish llms.txt (2,605 of 14,603)

Block rate by crawler, inside this segment robots.txt at the site root

CrawlerOperatorContent used forBlock rateBlockingMeasured
GPTBot OpenAI training 9.8% 1,434 14,602
CCBot Common Crawl Foundation archive 9.8% 1,434 14,602
Bytespider ByteDance training 9.8% 1,427 14,602
ClaudeBot Anthropic training 8.5% 1,248 14,602
Meta-ExternalAgent Meta training 7.9% 1,155 14,602
Applebot-Extended Apple training 7.9% 1,155 14,602
Google-Extended Google training 7.8% 1,134 14,602
Amazonbot Amazon training 7.7% 1,126 14,602
omgilibot Webz.io training 7.6% 1,107 14,602
anthropic-ai Anthropic training 7.6% 1,106 14,602
Diffbot Diffbot training 7.4% 1,078 14,602
cohere-ai Cohere live retrieval 7.2% 1,055 14,602
omgili Webz.io training 7.2% 1,052 14,602
Google-CloudVertexBot Google live retrieval 4.5% 660 14,602
GoogleOther Google training 4.4% 636 14,602
MistralAI-Training Mistral AI training 3.8% 556 14,602
SemrushBot-OCOB Semrush training 3.4% 497 14,602
FacebookBot Meta training 3.3% 489 14,602
PerplexityBot Perplexity answer index 3.3% 487 14,602
ChatGPT-User OpenAI live retrieval 3.3% 484 14,602
YouBot You.com answer index 3.0% 440 14,602
Timpibot Timpi training 2.9% 418 14,602
ImagesiftBot ImageSift (Hive) training 2.7% 391 14,602
Scrapy Zyte (open-source framework) training 2.6% 374 14,602
AI2Bot Allen Institute for AI training 2.4% 346 14,602
cohere-training-data-crawler Cohere training 2.1% 303 14,602
OAI-SearchBot OpenAI answer index 1.9% 280 14,602
Meta-ExternalFetcher Meta live retrieval 1.8% 267 14,602
MistralAI-User Mistral AI live retrieval 1.7% 247 14,602
Perplexity-User Perplexity live retrieval 1.7% 244 14,602
Claude-User Anthropic live retrieval 1.7% 241 14,602
DuckAssistBot DuckDuckGo live retrieval 1.6% 238 14,602
Meta-WebIndexer Meta answer index 1.6% 232 14,602
Claude-SearchBot Anthropic answer index 1.5% 222 14,602
PanguBot Huawei training 1.5% 219 14,602
Kangaroo Bot Kangaroo LLM training 1.5% 214 14,602
Applebot Apple answer index 1.3% 194 14,602
Amzn-SearchBot Amazon answer index 1.1% 164 14,602
TikTokSpider ByteDance live retrieval 0.8% 122 14,602
Amzn-User Amazon live retrieval 0.8% 122 14,602
MistralAI-Index Mistral AI answer index 0.4% 60 14,602
facebookexternalhit Meta live retrieval 0.2% 27 14,602
Meta-ExternalAds Meta training 0.1% 21 14,602
Googlebot-News Google answer index 0.1% 18 14,602
Bingbot Microsoft answer index 0.1% 12 14,602
OAI-AdsBot OpenAI live retrieval 0.0% 6 14,602
Googlebot Google answer index 0.0% 6 14,602
msnbot Microsoft answer index 0.0% 3 14,602

Block rate is the share of this segment's domains holding a policy record for that agent whose robots.txt disallows it at the site root, shown with its numerator and denominator. Denominators differ between agents because a record only exists once a domain has been measured for it. A domain serving no robots.txt counts as allowing everything, which is what the standard specifies.

Most accessible

DomainScoreEngines blockedllms.txt
1 axiomgraph.com 100 0 yes
2 quietnodes.com 100 0 yes
3 vwthemes.com 99 0 yes
4 motherearthnews.com 99 0 yes
5 alastyr.com 99 0 yes
6 reliaslearning.com 99 0 yes
7 dataimpulse.com 99 0 yes
8 webkul.com 99 0 yes
9 wpforms.com 99 0 yes
10 alidropship.com 99 0 yes
11 trackingplan.com 99 0 yes
12 infragistics.com 99 0 yes
13 azdigi.com 99 0 yes
14 lusha.com 99 0 yes
15 phorest.com 99 0 yes

Highest AI access scores among the 14,603 domains measured in this segment.

Least accessible

DomainScoreEngines blockedllms.txt
1 pivotalweather.com 25 5 no
2 singaporeair.com 26 0 no
3 maxwinexch99.com 27 2 no
4 dialmyapp.com 28 0 no
5 pghpedia.com 28 0 no
6 zvuk.com 29 0 no
7 digialm.com 29 0 no
8 evraz.com 29 0 no
9 rssing.com 29 4 no
10 itar-tass.com 29 0 no
11 flowwow.com 29 0 no
12 fstravel.com 29 0 no
13 easytrackbloodpressure.com 29 0 no
14 youdo.com 29 0 no
15 cloudtaxoffice.com 29 0 no

Lowest AI access scores among the same 14,603 domains.

Compare with the rest of the grouping top-level domain segments

The largest 12 other segments in this grouping, ordered by size. The share beside each is that segment's own answer-engine block rate over its own count. Full table on the segments index.

What this segment does not tell you read this before citing it

A top-level domain is a weak proxy for jurisdiction. Most of them are open to any registrant anywhere, so a domain under a country-code registry may be operated from outside that country by a company no local law reaches. Read these rows as differences between registration communities and the publishing cultures inside them, not as differences between legal regimes.

Measure your own domain

The audit that produced every number on this page runs on any domain in about six seconds: robots.txt resolved for 48 agents, then live requests sent as four of them to see whether the edge agrees with the file.