Live measurement, not robots.txt

Which edge refuses AI crawlers for you

A permissive robots.txt means nothing if the layer in front of your origin returns 403 to anything that looks like a bot. This page measures what actually happens: a real request sent as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot from a datacentre address, grouped by the edge provider identified from response headers.

Of the 7,733 sites where an edge provider could be identified, sites behind Sucuri refuse an AI user agent most often on repeat probes, 39.6 percent of 53 measured, against 18.7 percent across the corpus as a whole.

The measurement

Of 7,733 domains where the edge provider could be identified from response headers, 39.6% of the 53 sitting behind Sucuri refuse a request carrying an AI crawler user agent, against 18.7% across the corpus. Each refusal was reproduced on two consecutive independent scans from different locations; single-observation refusals are excluded because roughly one in eleven does not reproduce.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Which edge refuses AI crawlers", measured 2026-11-22. https://crawlcensus.com/edge

7,733
sites with an identified edge
990
confirmed refusals
13%
of identified sites
22,238
edge not identifiable
What "confirmed" means here. A single refusal is weak evidence: datacentre IP reputation, a momentary rate limit or a geographic rule can all produce one. A site is only counted as confirmed when it refused an AI user agent on two or more consecutive independent scans, run at different times from different edge locations. The single-scan column is shown beside it so the gap between the two is visible rather than hidden.
The false-positive rate, measured. Re-probing every site that refused on a first pass, roughly one in eleven did not refuse again. Those sites are excluded from the confirmed column. That is the error rate any single-probe study of AI crawler blocking carries silently, including the first pass of this one.

Refusal rate by edge provider confirmed on repeat probes

EdgeSitesConfirmed refusalRefused at least onceBlock an answer engine in robots.txtMean score
Sucuri 53 39.6%
21 of 53
43.4%
23 of 53
4% 76.3
LiteSpeed 340 29.7%
101 of 340
32.6%
111 of 340
12% 82.2
Akamai 787 24.0%
189 of 787
27.2%
214 of 787
11% 72.9
Kinsta 125 17.6%
22 of 125
20.0%
25 of 125
2% 87.4
Fastly 1,407 11.8%
166 of 1,407
13.5%
190 of 1,407
15% 77.1
CloudFront 3,475 10.9%
380 of 3,475
13.1%
455 of 3,475
12% 74.4
Vercel 693 8.8%
61 of 693
9.2%
64 of 693
5% 82.3
Azure Front Door 293 7.2%
21 of 293
9.2%
27 of 293
7% 75.2
Imperva 290 5.5%
16 of 290
6.6%
19 of 290
4% 66.4
Netlify 223 5.4%
12 of 223
5.4%
12 of 223
2% 81.3
Google Cloud 47 2.1%
1 of 47
2.1%
1 of 47
2% 63.8
bar chartSucuri (n=53)39.6%LiteSpeed (n=340)29.7%Akamai (n=787)24%Kinsta (n=125)17.6%Fastly (n=1407)11.8%CloudFront (n=3475)10.9%Vercel (n=693)8.8%Azure Front Door (n=293)7.2%Imperva (n=290)5.5%Netlify (n=223)5.4%Google Cloud (n=47)2.1%

How the edge is identified and why most cannot be

Only headers that survive the trip are trusted: x-vercel-id, x-amz-cf-id, fastly-restarts, x-akamai-request-id, x-nf-request-id, x-sucuri-id, x-iinfo, x-litespeed-cache and their siblings. The server header and cf-ray are deliberately ignored, because a request made from a Cloudflare Worker comes back carrying both regardless of what the origin actually runs. Trusting them would have reported every site on the web as Cloudflare-fronted, which is exactly the error this page exists to avoid making.

The consequence is honest but limited coverage: 22,238 measured sites expose no identifying header and are excluded entirely rather than guessed at. No provider is named unless its own header named it.

Full method on the methodology page. Underlying rows in data.json, field edge.

If your provider is on this list what to do

Being behind a provider with a high rate does not mean you are blocking anything. It usually means a managed bot rule set is enabled by default and nobody chose it deliberately. Check your own domain, then allow the retrieval agents you want citing you while leaving the rest as they are.

The report names every agent that was refused and the status it received. The guide covers detection and the fix per provider.