Which edge refuses AI crawlers for you
A permissive robots.txt means nothing if the layer in front of your origin returns 403 to anything that looks like a bot. This page measures what actually happens: a real request sent as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot from a datacentre address, grouped by the edge provider identified from response headers.
Of the 7,840 sites where an edge provider could be identified, sites behind Sucuri refuse an AI user agent most often on repeat probes, 36.5 percent of 52 measured, against 19.2 percent across the corpus as a whole.
Of 7,840 domains where the edge provider could be identified from response headers, 36.5% of the 52 sitting behind Sucuri refuse a request carrying an AI crawler user agent, against 19.2% across the corpus. Each refusal was reproduced on two consecutive independent scans from different locations; single-observation refusals are excluded because roughly one in eleven does not reproduce.
Measured 2026-10-08 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "Which edge refuses AI crawlers", measured 2026-10-08. https://crawlcensus.com/edge
Refusal rate by edge provider confirmed on repeat probes
| Edge | Sites | Confirmed refusal | Refused at least once | Block an answer engine in robots.txt | Mean score |
|---|---|---|---|---|---|
| Sucuri | 52 | 36.5% 19 of 52 |
40.4% 21 of 52 |
2% | 75.8 |
| LiteSpeed | 339 | 36.0% 122 of 339 |
38.3% 130 of 339 |
2% | 81.9 |
| Akamai | 814 | 24.9% 203 of 814 |
26.8% 218 of 814 |
11% | 72.9 |
| Kinsta | 121 | 19.8% 24 of 121 |
20.7% 25 of 121 |
0% | 87.7 |
| Fastly | 1,431 | 12.2% 175 of 1,431 |
14.3% 205 of 1,431 |
13% | 77.5 |
| CloudFront | 3,544 | 11.8% 419 of 3,544 |
14.8% 524 of 3,544 |
11% | 74.2 |
| Vercel | 686 | 8.2% 56 of 686 |
9.2% 63 of 686 |
3% | 81.6 |
| Azure Front Door | 291 | 6.9% 20 of 291 |
7.2% 21 of 291 |
8% | 75.5 |
| Netlify | 226 | 5.3% 12 of 226 |
5.8% 13 of 226 |
2% | 80.4 |
| Imperva | 289 | 4.2% 12 of 289 |
5.9% 17 of 289 |
3% | 67.9 |
| Google Cloud | 47 | 2.1% 1 of 47 |
2.1% 1 of 47 |
2% | 63.7 |
How the edge is identified and why most cannot be
Only headers that survive the trip are trusted: x-vercel-id, x-amz-cf-id, fastly-restarts, x-akamai-request-id, x-nf-request-id, x-sucuri-id, x-iinfo, x-litespeed-cache and their siblings. The server header and cf-ray are deliberately ignored, because a request made from a Cloudflare Worker comes back carrying both regardless of what the origin actually runs. Trusting them would have reported every site on the web as Cloudflare-fronted, which is exactly the error this page exists to avoid making.
The consequence is honest but limited coverage: 22,346 measured sites expose no identifying header and are excluded entirely rather than guessed at. No provider is named unless its own header named it.
Full method on the methodology page. Underlying rows in data.json, field edge.
If your provider is on this list what to do
Being behind a provider with a high rate does not mean you are blocking anything. It usually means a managed bot rule set is enabled by default and nobody chose it deliberately. Check your own domain, then allow the retrieval agents you want citing you while leaving the rest as they are.
The report names every agent that was refused and the status it received. The guide covers detection and the fix per provider.