Open dataset · CC BY 4.0

AI crawler access, segment by segment

Every domain counted below went through the same audit: read its robots.txt, resolve the rules for the 48 tracked AI agents, then send live requests as four of them and record what came back. No segment gets a different test and nothing is self-reported, so a rate in one table can be set against a rate in another without adjustment.

A segment only appears once enough of its members have been measured for the rate to hold still. Below roughly 40 domains a single robots.txt edit moves the number by tens of points, which is noise wearing the clothes of a finding. 85 segments across 3 groupings currently clear that bar, drawn from 29,997 measured domains.

Top-level domain 63 segments, 27,781 domains

Grouped by the registry a domain sits under. This is the closest thing the corpus has to a geographic view, and it separates the widest: block rates differ several-fold between country-code registries, a larger spread than any other cut of the same data produces.

A top-level domain is a weak proxy for jurisdiction. Most of them are open to any registrant anywhere, so a domain under a country-code registry may be operated from outside that country by a company no local law reaches. Read these rows as differences between registration communities and the publishing cultures inside them, not as differences between legal regimes.

Segment Sites Mean score Blocks an answer engine Blocks a training crawler Refuses AI at the edge Publishes llms.txt
.com 14,603 74.5 10% 1,430/14,603 15% 2,144/14,603 20% 2,926/14,603 18% 2,605/14,603
.org 1,840 73.4 9% 170/1,840 13% 247/1,840 22% 410/1,840 10% 183/1,840
.net 1,306 70.8 9% 118/1,306 12% 157/1,306 21% 272/1,306 11% 148/1,306
.ru 1,157 69.0 3% 33/1,157 6% 70/1,157 12% 136/1,157 9% 107/1,157
.de 823 77.2 17% 136/823 26% 213/823 13% 106/823 7% 58/823
.io 654 76.9 4% 25/654 6% 39/654 11% 70/654 30% 198/654
.edu 471 78.3 3% 14/471 7% 32/471 10% 48/471 7% 33/471
.uk 429 74.9 17% 75/429 24% 104/429 22% 95/429 10% 44/429
.jp 401 71.4 9% 37/401 17% 67/401 10% 40/401 6% 24/401
.br 360 75.6 8% 27/360 10% 36/360 19% 69/360 14% 52/360
.fr 320 75.9 17% 53/320 23% 72/320 15% 48/320 5% 16/320
.in 302 68.6 6% 17/302 9% 26/302 16% 48/302 11% 34/302
.it 267 75.1 13% 36/267 21% 55/267 22% 59/267 7% 20/267
.cn 247 63.0 2% 4/247 1% 3/247 2% 5/247 8% 19/247
.pl 239 74.5 8% 20/239 24% 57/239 16% 39/239 10% 23/239
.gov 208 69.9 4% 9/208 5% 10/208 21% 43/208 2% 5/208
.co 202 77.2 8% 17/202 14% 28/202 15% 31/202 28% 57/202
.ai 200 79.6 2% 3/200 7% 14/200 10% 19/200 46% 92/200
.nl 191 76.8 10% 20/191 17% 33/191 13% 24/191 15% 29/191
.tv 190 69.7 7% 14/190 12% 23/190 21% 39/190 4% 8/190
.es 167 75.0 20% 34/167 26% 44/167 16% 26/167 7% 11/167
.ca 159 74.8 7% 11/159 10% 16/159 24% 38/159 7% 11/159
.au 150 77.0 9% 14/150 15% 22/150 19% 28/150 10% 15/150
.app 132 76.1 5% 7/132 11% 15/132 10% 13/132 33% 43/132
.cz 130 73.7 9% 12/130 25% 32/130 12% 15/130 10% 13/130
.me 124 71.7 6% 7/124 9% 11/124 16% 20/124 14% 17/124
.id 118 69.5 8% 9/118 13% 15/118 14% 17/118 8% 10/118
.tr 118 72.3 8% 10/118 15% 18/118 11% 13/118 26% 31/118
.eu 110 74.7 5% 6/110 8% 9/110 20% 22/110 15% 16/110
.se 99 75.6 15% 15/99 18% 18/99 14% 14/99 12% 12/99
.ch 95 74.8 9% 9/95 19% 18/95 14% 13/95 9% 9/95
.vn 92 77.2 3% 3/92 4% 4/92 12% 11/92 24% 22/92
.ua 91 77.9 3% 3/91 13% 12/91 15% 14/91 20% 18/91
.info 87 70.0 11% 10/87 20% 17/87 29% 25/87 13% 11/87
.at 83 75.9 13% 11/83 22% 18/83 17% 14/83 10% 8/83
.ir 82 77.8 0% 0/82 0% 0/82 7% 6/82 10% 8/82
.hu 81 75.4 5% 4/81 11% 9/81 20% 16/81 7% 6/81
.gr 75 79.3 7% 5/75 13% 10/75 21% 16/75 9% 7/75
.dev 74 74.6 1% 1/74 5% 4/74 9% 7/74 34% 25/74
.be 73 73.5 8% 6/73 12% 9/73 16% 12/73 7% 5/73
.mx 71 71.1 10% 7/71 14% 10/71 25% 18/71 7% 5/71
.kr 70 67.9 13% 9/70 23% 16/70 17% 12/70 9% 6/70
.ro 65 78.4 6% 4/65 17% 11/65 15% 10/65 17% 11/65
.ar 64 70.7 6% 4/64 13% 8/64 28% 18/64 8% 5/64
.to 64 68.4 19% 12/64 20% 13/64 31% 20/64 13% 8/64
.cc 63 69.0 13% 8/63 19% 12/63 22% 14/63 11% 7/63
.cloud 63 66.8 2% 1/63 2% 1/63 11% 7/63 17% 11/63
.fi 59 75.7 19% 11/59 25% 15/59 7% 4/59 8% 5/59
.cl 57 76.9 7% 4/57 9% 5/57 16% 9/57 28% 16/57
.pro 57 70.4 11% 6/57 16% 9/57 18% 10/57 18% 10/57
.za 55 71.0 15% 8/55 22% 12/55 18% 10/55 16% 9/55
.no 52 75.0 31% 16/52 35% 18/52 6% 3/52 8% 4/52
.gg 49 70.0 22% 11/49 24% 12/49 33% 16/49 20% 10/49
.pt 49 70.8 14% 7/49 22% 11/49 27% 13/49 4% 2/49
.tw 49 68.4 14% 7/49 20% 10/49 18% 9/49 14% 7/49
.kz 47 71.9 2% 1/47 9% 4/47 11% 5/47 6% 3/47
.dk 46 73.2 20% 9/46 26% 12/46 11% 5/46 7% 3/46
.online 46 70.0 11% 5/46 11% 5/46 20% 9/46 4% 2/46
.us 44 69.8 9% 4/44 9% 4/44 16% 7/44 9% 4/44
.live 41 65.4 20% 8/41 24% 10/41 34% 14/41 0% 0/41
.my 40 68.9 13% 5/40 15% 6/40 25% 10/40 8% 3/40
.site 40 76.3 13% 5/40 15% 6/40 20% 8/40 15% 6/40
.xyz 40 67.8 13% 5/40 13% 5/40 28% 11/40 18% 7/40

Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 29,997-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.

bar chart.no (n=52)31%.gg (n=49)22%.es (n=167)20%.dk (n=46)20%.live (n=41)20%.to (n=64)19%.fi (n=59)19%.uk (n=429)17%.fr (n=320)17%.de (n=823)17%.se (n=99)15%.za (n=55)15%

Share of each registry's measured domains whose robots.txt disallows at least one answer engine at the site root. The count beside each label is that registry's denominator; registries with fewer than 40 measured domains are left out of this chart. Click a bar for the segment.

Publishing platform 11 segments, 6,007 domains

Grouped by the system that produced the page, read from the generator meta tag it left in the HTML. A platform's shipped defaults decide a site's crawler posture more often than the site owner does, so these rows measure defaults as much as intent.

Platform detection relies on a generator meta tag, so coverage is partial and skewed. A site that strips the tag, or a system that never writes one, is invisible here. These rows therefore over-represent platforms that advertise themselves and say nothing at all about the ones that do not.

Segment Sites Mean score Blocks an answer engine Blocks a training crawler Refuses AI at the edge Publishes llms.txt
wordpress 1,958 82.2 11% 225/1,958 14% 275/1,958 24% 461/1,958 16% 323/1,958
next.js 1,522 76.3 10% 147/1,522 16% 246/1,522 17% 265/1,522 19% 289/1,522
drupal 829 75.7 7% 54/829 10% 79/829 22% 179/829 7% 61/829
client-rendered shell 588 54.2 5% 28/588 8% 48/588 11% 63/588 6% 35/588
nuxt 544 76.1 8% 44/544 14% 75/544 13% 69/544 18% 99/544
astro 136 83.3 5% 7/136 8% 11/136 10% 14/136 40% 54/136
gatsby 126 75.6 2% 3/126 3% 4/126 6% 7/126 20% 25/126
hugo 106 76.9 4% 4/106 4% 4/106 6% 6/106 12% 13/106
webflow 96 82.3 7% 7/96 8% 8/96 10% 10/96 31% 30/96
wix 59 83.7 2% 1/59 2% 1/59 7% 4/59 86% 51/59
joomla 43 73.1 7% 3/43 9% 4/43 7% 3/43 0% 0/43

Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 29,997-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.

Edge provider 11 segments, 7,733 domains

Grouped by the CDN or WAF in front of the origin, read from identifying response headers. This is where robots.txt and reality part company: a managed bot rule set can refuse an AI agent at the edge while the file at /robots.txt still says yes.

Only providers that expose an identifying response header can be detected. Sites behind an edge that stays quiet, or behind an origin that strips those headers, are absent from the count, so this sample is not representative of any provider's customer base.

Segment Sites Mean score Blocks an answer engine Blocks a training crawler Refuses AI at the edge Publishes llms.txt
CloudFront 3,475 74.4 12% 404/3,475 18% 617/3,475 13% 455/3,475 15% 528/3,475
Fastly 1,407 77.1 15% 216/1,407 20% 284/1,407 14% 190/1,407 14% 193/1,407
Akamai 787 72.9 11% 88/787 16% 123/787 27% 214/787 12% 96/787
Vercel 693 82.3 5% 33/693 8% 56/693 9% 64/693 43% 297/693
LiteSpeed 340 82.2 12% 40/340 14% 48/340 33% 111/340 23% 78/340
Azure Front Door 293 75.2 7% 20/293 13% 38/293 9% 27/293 10% 30/293
Imperva 290 66.4 4% 11/290 8% 24/290 7% 19/290 12% 35/290
Netlify 223 81.3 2% 4/223 4% 10/223 5% 12/223 34% 75/223
Kinsta 125 87.4 2% 3/125 5% 6/125 20% 25/125 40% 50/125
Sucuri 53 76.3 4% 2/53 8% 4/53 43% 23/53 25% 13/53
Google Cloud 47 63.8 2% 1/47 2% 1/47 2% 1/47 9% 4/47

Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 29,997-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.

Measure a domain

A scan adds the domain to the corpus, and to whichever segments it belongs to. The full breakdown is in the open dataset.