Which networks refuse AI crawlers
A server can decline to say who runs it. It cannot decline to have an address. Every domain here was resolved to an IP, the IP mapped to its autonomous system, and the refusal rate computed per operator - so the figure covers origins that strip every identifying header.
Origin networks were resolved for 30,240 of 30,319 measured domains, against 7,971 identifiable from response headers alone. Among networks hosting at least 50 measured domains, SERVICEPIPE has the highest refusal rate at 71% of 76 domains, against 23.0% across all resolved networks.
Measured 2026-10-10 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "AI crawler refusal by origin network", measured 2026-10-10. https://crawlcensus.com/networks
Refusal rate by origin network 25 or more domains each
Every network measured the table
| Network | ASN | Domains | Refuse AI agents | Rate | Charge instead |
|---|---|---|---|---|---|
| SERVICEPIPE | AS201706 | 76 | 54 | 71% | - |
| Automattic | AS2635 | 289 | 110 | 38% | 22 |
| Cloudflare | AS13335 | 8,781 | 3,285 | 37% | 118 |
| Akamai | AS16625 | 885 | 278 | 31% | 2 |
| Datacamp | AS60068 | 76 | 13 | 17% | - |
| National Informatics Centre | AS4758 | 50 | 8 | 16% | - |
| Renater | AS789 | 51 | 8 | 16% | - |
| Amazon | AS8987 | 6,246 | 856 | 14% | 58 |
| AS15169 | 842 | 114 | 14% | - | |
| Fastly | AS54113 | 1,092 | 143 | 13% | 76 |
| DataWeb Global Group | AS39572 | 71 | 9 | 13% | - |
| Microsoft | AS8068 | 687 | 67 | 10% | 1 |
| DigitalOcean | AS14061 | 172 | 16 | 9% | - |
| OVH | AS16276 | 361 | 32 | 9% | - |
| Verein zur Foerderung eines Deutschen Fo | AS680 | 81 | 7 | 9% | - |
| Oracle | AS31898 | 64 | 5 | 8% | - |
| Hetzner | AS24940 | 464 | 36 | 8% | 2 |
| DDOS-GUARD | AS57724 | 149 | 11 | 7% | - |
| Imperva | AS19551 | 326 | 23 | 7% | - |
| Alibaba (US) Technology | AS45102 | 151 | 10 | 7% | - |
| Cloudflare London | AS209242 | 662 | 41 | 6% | - |
| HLL | AS51115 | 147 | 8 | 5% | - |
| Hangzhou Alibaba Advertising | AS37963 | 203 | 9 | 4% | - |
| Shenzhen Tencent Computer Systems Compan | AS45090 | 72 | 3 | 4% | - |
| JSC Selectel | AS49505 | 166 | 6 | 4% | - |
| IDC | AS23724 | 60 | 2 | 3% | - |
| Yandex.Cloud | AS200350 | 76 | 2 | 3% | - |
| Tencent Building | AS132203 | 50 | 1 | 2% | - |
| F5 Networks SARL | AS35280 | 52 | 1 | 2% | - |
| IONOS SE | AS8560 | 53 | 0 | 0% | - |
Why this differs from the edge table method
The edge provider table reports what a server says about itself: a server header, a cf-ray, an x-served-by. Roughly three quarters of measured domains say nothing at all, and a header can be forged or stripped.
This page reports something a site cannot suppress. The domain resolves to an address, the address belongs to an autonomous system, and that mapping is published in the global routing table. It answers a different question - not "which CDN is in front of this origin" but "whose network is this origin on" - and for a self-hosted origin those are the same answer, while for a CDN-fronted origin the anycast address belongs to the CDN.
Networks hosting fewer than 50 measured domains are excluded: a refusal rate over a handful of sites is noise. Resolution uses DNS over HTTPS and the Team Cymru IP-to-ASN service, both queried live at scan time.