Which networks refuse AI crawlers
A server can decline to say who runs it. It cannot decline to have an address. Every domain here was resolved to an IP, the IP mapped to its autonomous system, and the refusal rate computed per operator - so the figure covers origins that strip every identifying header.
Origin networks were resolved for 30,030 of 30,345 measured domains, against 7,804 identifiable from response headers alone. Among networks hosting at least 50 measured domains, SERVICEPIPE has the highest refusal rate at 72% of 74 domains, against 22.0% across all resolved networks.
Measured 2026-08-25 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "AI crawler refusal by origin network", measured 2026-08-25. https://crawlcensus.com/networks
Refusal rate by origin network 25 or more domains each
Every network measured the table
| Network | ASN | Domains | Refuse AI agents | Rate | Charge instead |
|---|---|---|---|---|---|
| SERVICEPIPE | AS201706 | 74 | 53 | 72% | - |
| Automattic | AS2635 | 288 | 108 | 38% | 22 |
| Cloudflare | AS13335 | 8,903 | 3,214 | 36% | 114 |
| Akamai | AS16625 | 819 | 223 | 27% | 1 |
| Datacamp | AS60068 | 67 | 12 | 18% | - |
| AS15169 | 855 | 118 | 14% | - | |
| Renater | AS789 | 51 | 7 | 14% | - |
| Amazon | AS8987 | 6,125 | 763 | 12% | 54 |
| Verein zur Foerderung eines Deutschen Fo | AS680 | 79 | 9 | 11% | - |
| Fastly | AS54113 | 1,092 | 124 | 11% | 62 |
| DigitalOcean | AS14061 | 165 | 18 | 11% | - |
| Microsoft | AS8068 | 674 | 67 | 10% | 1 |
| OVH | AS16276 | 351 | 26 | 7% | - |
| Hetzner | AS24940 | 453 | 33 | 7% | 1 |
| DDOS-GUARD | AS57724 | 156 | 11 | 7% | - |
| Oracle | AS31898 | 57 | 4 | 7% | - |
| Imperva | AS19551 | 329 | 22 | 7% | - |
| DataWeb Global Group | AS39572 | 123 | 8 | 7% | - |
| Alibaba (US) Technology | AS45102 | 154 | 10 | 6% | - |
| Cloudflare London | AS209242 | 681 | 43 | 6% | - |
| Yandex.Cloud | AS200350 | 80 | 5 | 6% | - |
| Optimizely AB | AS30811 | 50 | 3 | 6% | - |
| Shenzhen Tencent Computer Systems Compan | AS45090 | 69 | 4 | 6% | - |
| HLL | AS51115 | 150 | 8 | 5% | - |
| IDC | AS23724 | 58 | 3 | 5% | - |
| JSC Selectel | AS49505 | 154 | 6 | 4% | - |
| Hangzhou Alibaba Advertising | AS37963 | 200 | 6 | 3% | - |
| IONOS SE | AS8560 | 54 | 1 | 2% | - |
| F5 Networks SARL | AS35280 | 52 | 0 | 0% | - |
Why this differs from the edge table method
The edge provider table reports what a server says about itself: a server header, a cf-ray, an x-served-by. Roughly three quarters of measured domains say nothing at all, and a header can be forged or stripped.
This page reports something a site cannot suppress. The domain resolves to an address, the address belongs to an autonomous system, and that mapping is published in the global routing table. It answers a different question - not "which CDN is in front of this origin" but "whose network is this origin on" - and for a self-hosted origin those are the same answer, while for a CDN-fronted origin the anycast address belongs to the CDN.
Networks hosting fewer than 50 measured domains are excluded: a refusal rate over a handful of sites is noise. Resolution uses DNS over HTTPS and the Team Cymru IP-to-ASN service, both queried live at scan time.