Crawl Census
Scan a site
Open dataset · CC BY 4.0

AI crawler access, segment by segment

Every domain counted below went through the same audit: read its robots.txt, resolve the rules for the 48 tracked AI agents, then send live requests as four of them and record what came back. No segment gets a different test and nothing is self-reported, so a rate in one table can be set against a rate in another without adjustment.

A segment only appears once enough of its members have been measured for the rate to hold still. Below roughly 40 domains a single robots.txt edit moves the number by tens of points, which is noise wearing the clothes of a finding. 84 segments across 3 groupings currently clear that bar, drawn from 30,226 measured domains.

Top-level domain 62 segments, 27,914 domains

Grouped by the registry a domain sits under. This is the closest thing the corpus has to a geographic view, and it separates the widest: block rates differ several-fold between country-code registries, a larger spread than any other cut of the same data produces.

A top-level domain is a weak proxy for jurisdiction. Most of them are open to any registrant anywhere, so a domain under a country-code registry may be operated from outside that country by a company no local law reaches. Read these rows as differences between registration communities and the publishing cultures inside them, not as differences between legal regimes.

Segment Sites Mean score Blocks an answer engine Blocks a training crawler Refuses AI at the edge Publishes llms.txt
.com 14,804 74.2 7% 1,022/14,804 12% 1,778/14,804 21% 3,067/14,804 17% 2,481/14,804
.org 1,670 72.7 6% 92/1,670 11% 177/1,670 25% 410/1,670 9% 147/1,670
.net 1,305 70.7 4% 58/1,305 8% 107/1,305 22% 287/1,305 10% 136/1,305
.ru 1,167 69.2 3% 38/1,167 8% 90/1,167 10% 122/1,167 9% 108/1,167
.de 827 77.1 16% 130/827 25% 208/827 13% 110/827 7% 56/827
.io 660 76.7 2% 12/660 4% 28/660 10% 68/660 29% 191/660
.edu 473 78.3 2% 10/473 6% 27/473 10% 48/473 7% 32/473
.uk 446 75.3 17% 74/446 24% 106/446 22% 98/446 9% 40/446
.jp 418 71.2 9% 36/418 16% 66/418 10% 42/418 5% 22/418
.br 376 75.6 6% 22/376 8% 30/376 19% 71/376 14% 51/376
.fr 330 75.6 15% 51/330 22% 73/330 15% 48/330 3% 11/330
.in 304 68.0 3% 9/304 6% 18/304 15% 47/304 10% 29/304
.it 272 74.8 15% 42/272 23% 62/272 23% 62/272 6% 15/272
.cn 250 62.1 2% 4/250 0.8% 2/250 3% 8/250 7% 17/250
.pl 245 74.8 7% 18/245 22% 55/245 16% 38/245 10% 24/245
.gov 225 70.0 4% 8/225 5% 11/225 22% 49/225 2% 5/225
.co 206 76.6 5% 10/206 10% 21/206 18% 37/206 24% 50/206
.ai 199 79.2 0% 0/199 6% 12/199 10% 19/199 44% 88/199
.tv 193 70.4 4% 8/193 10% 19/193 21% 40/193 4% 7/193
.nl 188 76.6 9% 16/188 15% 29/188 13% 25/188 14% 26/188
.es 169 75.2 18% 31/169 24% 41/169 21% 35/169 7% 11/169
.ca 161 74.2 5% 8/161 7% 12/161 27% 44/161 6% 9/161
.au 153 77.0 8% 12/153 13% 20/153 18% 28/153 8% 12/153
.cz 132 73.8 7% 9/132 23% 30/132 14% 18/132 9% 12/132
.me 126 71.4 2% 2/126 6% 7/126 15% 19/126 13% 16/126
.app 124 74.8 2% 2/124 8% 10/124 10% 13/124 25% 31/124
.id 121 69.9 4% 5/121 8% 10/121 15% 18/121 11% 13/121
.tr 118 73.0 5% 6/118 14% 17/118 7% 8/118 27% 32/118
.eu 107 74.0 5% 5/107 8% 9/107 20% 21/107 11% 12/107
.ch 99 74.8 6% 6/99 18% 18/99 14% 14/99 8% 8/99
.se 98 75.9 14% 14/98 18% 18/98 13% 13/98 7% 7/98
.vn 92 77.3 2% 2/92 4% 4/92 11% 10/92 24% 22/92
.ua 91 78.3 3% 3/91 13% 12/91 15% 14/91 19% 17/91
.info 86 69.6 6% 5/86 14% 12/86 29% 25/86 15% 13/86
.at 82 76.6 12% 10/82 22% 18/82 18% 15/82 7% 6/82
.hu 80 75.4 5% 4/80 10% 8/80 18% 14/80 8% 6/80
.ir 79 77.2 0% 0/79 1% 1/79 1% 1/79 9% 7/79
.be 75 74.5 8% 6/75 12% 9/75 17% 13/75 8% 6/75
.dev 75 73.0 1% 1/75 5% 4/75 12% 9/75 32% 24/75
.gr 74 77.8 7% 5/74 12% 9/74 22% 16/74 8% 6/74
.kr 73 68.4 12% 9/73 26% 19/73 14% 10/73 5% 4/73
.mx 72 71.1 13% 9/72 17% 12/72 25% 18/72 6% 4/72
.ro 69 77.8 3% 2/69 13% 9/69 16% 11/69 16% 11/69
.ar 68 71.1 4% 3/68 9% 6/68 26% 18/68 9% 6/68
.to 64 69.7 5% 3/64 6% 4/64 30% 19/64 13% 8/64
.cloud 63 65.7 0% 0/63 0% 0/63 14% 9/63 14% 9/63
.cc 62 69.2 5% 3/62 13% 8/62 24% 15/62 10% 6/62
.fi 60 76.1 17% 10/60 25% 15/60 8% 5/60 7% 4/60
.za 60 71.2 13% 8/60 20% 12/60 18% 11/60 13% 8/60
.cl 57 77.2 4% 2/57 4% 2/57 18% 10/57 26% 15/57
.pro 53 68.2 0% 0/53 8% 4/53 19% 10/53 8% 4/53
.pt 50 72.3 10% 5/50 20% 10/50 24% 12/50 2% 1/50
.no 49 75.5 24% 12/49 29% 14/49 8% 4/49 6% 3/49
.online 49 69.1 2% 1/49 6% 3/49 24% 12/49 6% 3/49
.gg 48 68.8 15% 7/48 17% 8/48 33% 16/48 15% 7/48
.kz 48 70.8 4% 2/48 8% 4/48 6% 3/48 6% 3/48
.tw 48 67.3 13% 6/48 15% 7/48 25% 12/48 10% 5/48
.dk 46 74.2 17% 8/46 24% 11/46 7% 3/46 9% 4/46
.live 46 66.0 2% 1/46 7% 3/46 30% 14/46 0% 0/46
.us 45 68.2 11% 5/45 11% 5/45 22% 10/45 11% 5/45
.xyz 43 67.5 0% 0/43 0% 0/43 26% 11/43 19% 8/43
.my 41 67.2 2% 1/41 7% 3/41 22% 9/41 7% 3/41

Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 30,226-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.

bar chart.no (n=49)24%.es (n=169)18%.dk (n=46)17%.fi (n=60)17%.uk (n=446)17%.de (n=827)16%.fr (n=330)15%.it (n=272)15%.gg (n=48)15%.se (n=98)14%.za (n=60)13%.mx (n=72)13%

Share of each registry's measured domains whose robots.txt disallows at least one answer engine at the site root. The count beside each label is that registry's denominator; registries with fewer than 40 measured domains are left out of this chart. Click a bar for the segment.

Publishing platform 11 segments, 6,071 domains

Grouped by the system that produced the page, read from the generator meta tag it left in the HTML. A platform's shipped defaults decide a site's crawler posture more often than the site owner does, so these rows measure defaults as much as intent.

Platform detection relies on a generator meta tag, so coverage is partial and skewed. A site that strips the tag, or a system that never writes one, is invisible here. These rows therefore over-represent platforms that advertise themselves and say nothing at all about the ones that do not.

Segment Sites Mean score Blocks an answer engine Blocks a training crawler Refuses AI at the edge Publishes llms.txt
wordpress 1,930 82.1 6% 117/1,930 9% 171/1,930 26% 495/1,930 17% 326/1,930
next.js 1,600 76.0 8% 131/1,600 14% 230/1,600 18% 280/1,600 17% 278/1,600
drupal 842 75.9 5% 40/842 8% 68/842 22% 184/842 7% 58/842
client-rendered shell 617 54.3 3% 18/617 6% 36/617 10% 63/617 6% 35/617
nuxt 536 75.6 6% 31/536 11% 61/536 14% 77/536 17% 92/536
astro 134 82.5 3% 4/134 6% 8/134 9% 12/134 34% 45/134
gatsby 132 75.5 2% 2/132 2% 3/132 7% 9/132 21% 28/132
hugo 102 77.0 1.0% 1/102 1.0% 1/102 5% 5/102 11% 11/102
webflow 95 81.6 6% 6/95 8% 8/95 7% 7/95 26% 25/95
joomla 44 74.1 7% 3/44 7% 3/44 2% 1/44 0% 0/44
wix low n 39 84.6 3% 1/39 3% 1/39 3% 1/39 74% 29/39

Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 30,226-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.

Edge provider 11 segments, 7,819 domains

Grouped by the CDN or WAF in front of the origin, read from identifying response headers. This is where robots.txt and reality part company: a managed bot rule set can refuse an AI agent at the edge while the file at /robots.txt still says yes.

Only providers that expose an identifying response header can be detected. Sites behind an edge that stays quiet, or behind an origin that strips those headers, are absent from the count, so this sample is not representative of any provider's customer base.

Segment Sites Mean score Blocks an answer engine Blocks a training crawler Refuses AI at the edge Publishes llms.txt
CloudFront 3,534 74.2 11% 401/3,534 17% 614/3,534 15% 517/3,534 14% 499/3,534
Fastly 1,426 77.5 13% 186/1,426 18% 258/1,426 14% 204/1,426 14% 196/1,426
Akamai 813 72.9 11% 90/813 16% 127/813 27% 221/813 12% 94/813
Vercel 685 81.5 3% 22/685 7% 46/685 9% 62/685 41% 278/685
LiteSpeed 335 81.9 2% 8/335 4% 15/335 38% 128/335 22% 73/335
Azure Front Door 292 75.4 8% 24/292 14% 41/292 7% 21/292 9% 26/292
Imperva 287 67.7 3% 8/287 8% 23/287 6% 17/287 12% 34/287
Netlify 226 80.4 2% 4/226 4% 9/226 6% 13/226 31% 71/226
Kinsta 122 87.6 0.8% 1/122 4% 5/122 21% 26/122 45% 55/122
Sucuri 52 75.3 2% 1/52 8% 4/52 44% 23/52 21% 11/52
Google Cloud 47 63.6 2% 1/47 2% 1/47 2% 1/47 6% 3/47

Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 30,226-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.

Measure a domain

A scan adds the domain to the corpus, and to whichever segments it belongs to. The full breakdown is in the open dataset.