Which edge refuses AI crawlers for you
A permissive robots.txt means nothing if the layer in front of your origin returns 403 to anything that looks like a bot. This page measures what actually happens: a real request sent as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot from a datacentre address, grouped by the edge provider identified from response headers.
Of the 7,572 sites where an edge provider could be identified, sites behind LiteSpeed refuse an AI user agent most often on repeat probes, 29.1 percent of 354 measured, against 18.8 percent across the corpus as a whole.
Of 7,572 domains where the edge provider could be identified from response headers, 29.1% of the 354 sitting behind LiteSpeed refuse a request carrying an AI crawler user agent, against 18.8% across the corpus. Each refusal was reproduced on two consecutive independent scans from different locations; single-observation refusals are excluded because roughly one in eleven does not reproduce.
Measured 2026-08-24 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "Which edge refuses AI crawlers", measured 2026-08-24. https://crawlcensus.com/edge
Refusal rate by edge provider confirmed on repeat probes
| Edge | Sites | Confirmed refusal | Refused at least once | Block an answer engine in robots.txt | Mean score |
|---|---|---|---|---|---|
| LiteSpeed | 354 | 29.1% 103 of 354 |
35.9% 127 of 354 |
1% | 79.2 |
| Akamai | 767 | 18.9% 145 of 767 |
25.4% 195 of 767 |
9% | 72.9 |
| Sucuri | 51 | 15.7% 8 of 51 |
58.8% 30 of 51 |
2% | 75.0 |
| Kinsta | 122 | 9.8% 12 of 122 |
16.4% 20 of 122 |
2% | 87.7 |
| Fastly | 1,404 | 8.1% 114 of 1,404 |
14.3% 201 of 1,404 |
14% | 76.4 |
| CloudFront | 3,387 | 7.1% 239 of 3,387 |
13.5% 456 of 3,387 |
12% | 73.5 |
| Vercel | 647 | 5.1% 33 of 647 |
8.7% 56 of 647 |
3% | 79.7 |
| Azure Front Door | 288 | 3.8% 11 of 288 |
7.6% 22 of 288 |
8% | 74.4 |
| Netlify | 222 | 3.2% 7 of 222 |
5.4% 12 of 222 |
0.9% | 80.0 |
| Google Cloud | 45 | 2.2% 1 of 45 |
2.2% 1 of 45 |
2% | 62.0 |
| Imperva | 285 | 1.8% 5 of 285 |
6.3% 18 of 285 |
3% | 67.9 |
How the edge is identified and why most cannot be
Only headers that survive the trip are trusted: x-vercel-id, x-amz-cf-id, fastly-restarts, x-akamai-request-id, x-nf-request-id, x-sucuri-id, x-iinfo, x-litespeed-cache and their siblings. The server header and cf-ray are deliberately ignored, because a request made from a Cloudflare Worker comes back carrying both regardless of what the origin actually runs. Trusting them would have reported every site on the web as Cloudflare-fronted, which is exactly the error this page exists to avoid making.
The consequence is honest but limited coverage: 21,682 measured sites expose no identifying header and are excluded entirely rather than guessed at. No provider is named unless its own header named it.
Full method on the methodology page. Underlying rows in data.json, field edge.
If your provider is on this list what to do
Being behind a provider with a high rate does not mean you are blocking anything. It usually means a managed bot rule set is enabled by default and nobody chose it deliberately. Check your own domain, then allow the retrieval agents you want citing you while leaving the rest as they are.
The report names every agent that was refused and the status it received. The guide covers detection and the fix per provider.