AI crawler access, segment by segment
Every domain counted below went through the same audit: read its robots.txt, resolve the rules for the 48 tracked AI agents, then send live requests as four of them and record what came back. No segment gets a different test and nothing is self-reported, so a rate in one table can be set against a rate in another without adjustment.
A segment only appears once enough of its members have been measured for the rate to hold still. Below roughly 40 domains a single robots.txt edit moves the number by tens of points, which is noise wearing the clothes of a finding. 53 segments across 3 groupings currently clear that bar, drawn from 13,154 measured domains.
Top-level domain 34 segments, 11,810 domains
Grouped by the registry a domain sits under. This is the closest thing the corpus has to a geographic view, and it separates the widest: block rates differ several-fold between country-code registries, a larger spread than any other cut of the same data produces.
A top-level domain is a weak proxy for jurisdiction. Most of them are open to any registrant anywhere, so a domain under a country-code registry may be operated from outside that country by a company no local law reaches. Read these rows as differences between registration communities and the publishing cultures inside them, not as differences between legal regimes.
| Segment | Sites | Mean score | Blocks an answer engine | Blocks a training crawler | Refuses AI at the edge | Publishes llms.txt |
|---|---|---|---|---|---|---|
| .com | 6,762 | 71.6 | 8% 531/6,762 | 17% 1,141/6,762 | 21% 1,397/6,762 | 17% 1,120/6,762 |
| .org | 748 | 69.6 | 5% 39/748 | 15% 112/748 | 23% 175/748 | 8% 58/748 |
| .net | 593 | 68.1 | 6% 38/593 | 17% 99/593 | 21% 122/593 | 10% 58/593 |
| .ru | 468 | 66.8 | 3% 15/468 | 8% 36/468 | 9% 42/468 | 10% 46/468 |
| .de | 321 | 74.0 | 17% 55/321 | 29% 92/321 | 15% 48/321 | 7% 22/321 |
| .io | 308 | 72.7 | 2% 7/308 | 7% 21/308 | 8% 24/308 | 25% 78/308 |
| .jp | 236 | 68.1 | 7% 17/236 | 14% 33/236 | 9% 22/236 | 5% 12/236 |
| .edu | 216 | 74.3 | 2% 5/216 | 6% 12/216 | 9% 20/216 | 4% 9/216 |
| .br | 174 | 72.9 | 5% 9/174 | 13% 22/174 | 21% 37/174 | 12% 21/174 |
| .uk | 158 | 71.6 | 16% 25/158 | 28% 45/158 | 28% 44/158 | 7% 11/158 |
| .in | 151 | 64.5 | 2% 3/151 | 9% 14/151 | 18% 27/151 | 11% 16/151 |
| .gov | 150 | 68.8 | 3% 4/150 | 6% 9/150 | 23% 35/150 | 0.7% 1/150 |
| .fr | 142 | 72.4 | 25% 36/142 | 34% 48/142 | 15% 22/142 | 1% 2/142 |
| .pl | 118 | 72.4 | 10% 12/118 | 28% 33/118 | 14% 16/118 | 7% 8/118 |
| .it | 111 | 72.7 | 18% 20/111 | 28% 31/111 | 23% 26/111 | 4% 4/111 |
| .tv | 102 | 67.0 | 2% 2/102 | 21% 21/102 | 28% 29/102 | 4% 4/102 |
| .cn | 92 | 59.1 | 0% 0/92 | 0% 0/92 | 2% 2/92 | 2% 2/92 |
| .co | 92 | 72.9 | 4% 4/92 | 15% 14/92 | 22% 20/92 | 16% 15/92 |
| .ai | 86 | 74.0 | 1% 1/86 | 12% 10/86 | 15% 13/86 | 43% 37/86 |
| .ca | 80 | 72.7 | 9% 7/80 | 14% 11/80 | 26% 21/80 | 6% 5/80 |
| .nl | 71 | 72.1 | 10% 7/71 | 18% 13/71 | 15% 11/71 | 8% 6/71 |
| .es | 63 | 70.7 | 19% 12/63 | 32% 20/63 | 16% 10/63 | 13% 8/63 |
| .me | 60 | 70.7 | 3% 2/60 | 8% 5/60 | 18% 11/60 | 12% 7/60 |
| .cz | 55 | 69.6 | 9% 5/55 | 36% 20/55 | 18% 10/55 | 11% 6/55 |
| .tr | 54 | 67.4 | 13% 7/54 | 19% 10/54 | 4% 2/54 | 24% 13/54 |
| .au | 49 | 74.5 | 14% 7/49 | 18% 9/49 | 16% 8/49 | 10% 5/49 |
| .ch | 49 | 68.8 | 12% 6/49 | 27% 13/49 | 20% 10/49 | 6% 3/49 |
| .app | 46 | 72.7 | 0% 0/46 | 11% 5/46 | 11% 5/46 | 22% 10/46 |
| .vn | 46 | 73.2 | 4% 2/46 | 11% 5/46 | 11% 5/46 | 22% 10/46 |
| .gr | 45 | 77.8 | 11% 5/45 | 18% 8/45 | 22% 10/45 | 9% 4/45 |
| .id | 43 | 63.8 | 2% 1/43 | 7% 3/43 | 14% 6/43 | 5% 2/43 |
| .eu | 41 | 71.6 | 2% 1/41 | 7% 3/41 | 27% 11/41 | 10% 4/41 |
| .ir | 40 | 72.7 | 0% 0/40 | 0% 0/40 | 5% 2/40 | 10% 4/40 |
| .se | 40 | 70.2 | 18% 7/40 | 20% 8/40 | 18% 7/40 | 10% 4/40 |
Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 13,154-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.
Share of each registry's measured domains whose robots.txt disallows at least one answer engine at the site root. The count beside each label is that registry's denominator; registries with fewer than 40 measured domains are left out of this chart. Click a bar for the segment.
Publishing platform 9 segments, 2,752 domains
Grouped by the system that produced the page, read from the generator meta tag it left in the HTML. A platform's shipped defaults decide a site's crawler posture more often than the site owner does, so these rows measure defaults as much as intent.
Platform detection relies on a generator meta tag, so coverage is partial and skewed. A site that strips the tag, or a system that never writes one, is invisible here. These rows therefore over-represent platforms that advertise themselves and say nothing at all about the ones that do not.
| Segment | Sites | Mean score | Blocks an answer engine | Blocks a training crawler | Refuses AI at the edge | Publishes llms.txt |
|---|---|---|---|---|---|---|
| next.js | 832 | 72.4 | 9% 78/832 | 16% 131/832 | 18% 148/832 | 16% 131/832 |
| wordpress | 828 | 80.0 | 7% 60/828 | 20% 164/828 | 29% 243/828 | 16% 129/828 |
| drupal | 377 | 73.6 | 4% 16/377 | 11% 41/377 | 20% 77/377 | 6% 21/377 |
| client-rendered shell | 267 | 52.1 | 3% 8/267 | 13% 35/267 | 12% 32/267 | 5% 14/267 |
| nuxt | 224 | 70.7 | 8% 17/224 | 17% 39/224 | 17% 39/224 | 19% 43/224 |
| gatsby | 76 | 74.4 | 1% 1/76 | 4% 3/76 | 9% 7/76 | 25% 19/76 |
| astro | 54 | 77.8 | 2% 1/54 | 4% 2/54 | 7% 4/54 | 31% 17/54 |
| hugo | 54 | 71.9 | 2% 1/54 | 6% 3/54 | 4% 2/54 | 9% 5/54 |
| webflow | 40 | 77.9 | 18% 7/40 | 10% 4/40 | 15% 6/40 | 28% 11/40 |
Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 13,154-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.
Edge provider 10 segments, 8,827 domains
Grouped by the CDN or WAF in front of the origin, read from identifying response headers. This is where robots.txt and reality part company: a managed bot rule set can refuse an AI agent at the edge while the file at /robots.txt still says yes.
Only providers that expose an identifying response header can be detected. Sites behind an edge that stays quiet, or behind an origin that strips those headers, are absent from the count, so this sample is not representative of any provider's customer base.
| Segment | Sites | Mean score | Blocks an answer engine | Blocks a training crawler | Refuses AI at the edge | Publishes llms.txt |
|---|---|---|---|---|---|---|
| Cloudflare | 7,102 | 70.4 | 9% 640/7,102 | 18% 1,255/7,102 | 20% 1,407/7,102 | 13% 955/7,102 |
| CloudFront | 806 | 71.6 | 9% 70/806 | 14% 116/806 | 13% 108/806 | 14% 109/806 |
| Fastly | 320 | 73.6 | 13% 43/320 | 22% 70/320 | 15% 49/320 | 11% 36/320 |
| Akamai | 164 | 71.5 | 9% 15/164 | 14% 23/164 | 23% 38/164 | 12% 20/164 |
| Vercel | 141 | 76.1 | 4% 6/141 | 8% 11/141 | 12% 17/141 | 32% 45/141 |
| LiteSpeed | 96 | 78.6 | 0% 0/96 | 26% 25/96 | 41% 39/96 | 19% 18/96 |
| Imperva | 70 | 62.1 | 3% 2/70 | 4% 3/70 | 6% 4/70 | 4% 3/70 |
| Azure Front Door | 56 | 69.8 | 2% 1/56 | 11% 6/56 | 7% 4/56 | 7% 4/56 |
| Netlify | 45 | 77.7 | 2% 1/45 | 9% 4/45 | 4% 2/45 | 27% 12/45 |
| Kinsta low n | 27 | 87.5 | 4% 1/27 | 7% 2/27 | 19% 5/27 | 52% 14/27 |
Every percentage in this table is a share of that segment's own measured domains, shown beside it as a count over a denominator, never a share of the 13,154-domain corpus. Rows are ordered by size. A segment marked low n has fewer than 40 measured domains, which is too few for its rate to survive a single site's edit.
Measure a domain
A scan adds the domain to the corpus, and to whichever segments it belongs to. The full breakdown is in the open dataset.