AI crawler blocking at the edge: detect it and fix it

The most common AI crawl failure is not a robots.txt mistake. It is a permissive robots.txt sitting in front of an edge that answers 403, 429 or a JavaScript challenge to anything that looks automated. Retrieval fetchers do not solve interstitials. The assistant records a fetch failure and answers from another source, and nothing in your robots.txt or your analytics tells you it happened.

This failure is worth 6 points in the Reach dimension of a Crawl Census report for a reason: it is silent, it is common, and it inverts the intent of every other configuration decision you made.

Detect it in four minutes

Fetch your own home page as a browser, then as each crawler, and compare status code and body size. Anything that diverges from the baseline is your answer.

# Baseline: what a person gets.
curl -sS -o /dev/null -D - \
  -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36' \
  https://example.com/

# OpenAI retrieval fetcher: decides ChatGPT search citations.
curl -sS -o /dev/null -D - \
  -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot' \
  https://example.com/

# OpenAI training crawler.
curl -sS -o /dev/null -D - \
  -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' \
  https://example.com/

# Perplexity index crawler.
curl -sS -o /dev/null -D - \
  -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)' \
  https://example.com/

# Apple: Spotlight, Siri, Safari search and Apple Intelligence grounding.
curl -sS -o /dev/null -D - \
  -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)' \
  https://example.com/

# Common Crawl: the substrate for many training corpora.
curl -sS -o /dev/null -D - \
  -A 'CCBot/2.0 (https://commoncrawl.org/faq/)' \
  https://example.com/

For a compact comparison, print status and size only:

for ua in \
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36' \
  'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' \
  'CCBot/2.0 (https://commoncrawl.org/faq/)'
do
  printf '%s  ' "$(curl -sS -o /dev/null -w '%{http_code} %{size_download}' -A "$ua" https://example.com/)"
  printf '%s\n' "$ua"
done

Two caveats on the strings themselves. Anthropic has never published a full user-agent header for ClaudeBot, Claude-User or Claude-SearchBot; its support article documents the tokens, their purposes, Crawl-delay support and an IP list, and nothing more. The widely circulated ClaudeBot string is unverified against any Anthropic page, so probe with a string containing the token substring and read the result as indicative. Vendors that publish a version placeholder such as W.X.Y.Z, including Google and Amazon, should be matched by wildcard, never by exact version.

Probe /robots.txt as well as /. If the file itself is challenged, crawlers cannot read the permissions you wrote. Amazon documents that if it cannot fetch robots.txt it "will behave as if it does not exist", and Amazon caches the file for up to 30 days, so a transient challenge can shape behaviour for a month.

Read your own edge logs

The probe tells you the current state; logs tell you what has been happening. On Cloudflare, the presence of a cf-mitigated response header means the request was stopped by a WAF rule rather than by your origin. Cloudflare's own troubleshooting guidance puts it plainly: if the header is present, "your request was blocked by one of your WAF rules", and Security Events will name which one.

AI Crawl Control adds two views worth checking before you touch any configuration. Its Metrics tab groups crawler requests by crawler, category, operator, host and status code, so a wall of 403s to one operator is visible directly. Its Robots.txt tab reports the health and status code of your robots.txt per hostname and lists crawlers that requested explicitly disallowed paths, with the violated path and the specific directive.

What the AI Crawl Control block toggle actually does

Cloudflare documents the mechanism, and the details matter if you are debugging.

• Block is a WAF custom rule. Blocking AI crawlers through AI Crawl Control uses one WAF custom rule to block the selection you chose. Allowing all AI crawlers uses no custom rule. That rule counts against your plan's custom-rule allowance.
• Order of precedence. Traffic hits WAF custom rules, including AI Crawl Control's crawler blocks, then Cloudflare's bot solutions, then pay per crawl. Cloudflare's own advice is that if you intend to manage AI crawlers through AI Crawl Control, you should modify pre-existing WAF custom rules so they do not also act on AI crawlers.
• Detection quality varies by plan. On the free plan, AI Crawl Control identifies AI crawlers by user-agent string, which detects well-known self-identifying crawlers and nothing else.
• The block response is configurable, so what your probe sees depends on how the zone was set up.
• Managed robots.txt is not enforcement. Cloudflare's own documentation says of it: "robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level." If you turned that setting on expecting enforcement, you got a request instead.

Why legitimate fetchers get caught by default

This is not usually anybody's mistake. It is what generic bot detection is built to do.

Cloudflare's Bot Management assigns every request a bot score from 1 to 99, where 1 to 29 is likely automated. A legitimate answer-engine fetcher earns a low score by construction: it arrives from datacentre IP space, presents no browser fingerprint, runs no JavaScript, carries no cookies and no history. Every heuristic that catches a scraper catches it too. Super Bot Fight Mode's Definitely automated action, the Block AI bots setting in Bot Management, and any custom rule of the form "bot score below 30, not a verified bot, challenge" will all fire on OAI-SearchBot unless something exempts it first.

The exemption that matters is Cloudflare's verified-bot signal (cf.client.bot in expressions, cf.bot_management.verified_bot with Bot Management). Cloudflare maintains its allowlist "primarily through reverse DNS validation, ensuring the source IP matches the requesting service", plus ASN blocks and public lists. A Skip rule matching verified bots, placed first, is what keeps a legitimate crawler out of your blocking rules. Without it, ordering alone will bite you.

The scale of what is at stake is worth one number. Cloudflare's crawl-to-refer analysis for 19–26 June 2025 found ratios from Anthropic at 70,900:1 down to Mistral at 0.1:1, and Cloudflare states the ratios "may overstate" the figures because native-app referrals send no Referer header. Crawl volume dwarfs referral volume for almost every platform. That is an argument for making the block deliberate, in either direction, rather than accidental.

Allow specific agents safely

User agents are trivially forged. Common Crawl says so directly: "we are aware of crawlers falsely identifying themselves as CCBot." Match the user agent to narrow the candidate set, then verify before granting anything. Every operator below publishes machine-readable IP ranges, and several publish reverse-DNS domains.

Where reverse DNS exists, the check is a forward-confirmed lookup: resolve the client IP to a hostname, confirm the hostname ends in the operator's domain, then resolve that hostname back to an address and confirm it matches the client IP. Skipping the forward confirmation makes the check worthless, because PTR records live with whoever controls the address block.

# Forward-confirmed reverse DNS, by hand.
host 66.249.66.1
# -> 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
# -> crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Order the rules so verification happens before blocking. On Cloudflare that means a Skip rule for verified bots placed first, then your IP-list allow rules for operators Cloudflare does not verify, then your blocking rules. A blocking rule that runs before the allowlist is an allowlist that does nothing.

Three things not to do

1. Do not allowlist on user agent alone. It is the one control an attacker can set for free, and the operators tell you it is being abused.
2. Do not match the substring "bot" in a blocking rule. It catches Googlebot, Applebot, PerplexityBot, bingbot and every retrieval fetcher you intended to allow, which is exactly how most of the blocks we detect got there.
3. Do not leave /robots.txt behind a challenge. A crawler that cannot read your permissions does not default to caution in any consistent way. Amazon documents behaving as though the file does not exist, and holds that state for up to 30 days.

One last calibration. Known Agents puts robots.txt effectiveness, meaning how often bots follow the rules, at 96.4% across its measured sites as of 2026-08-21 — a vendor-published figure from a self-selected sample of sites that already care about bot traffic, so treat it as indicative rather than as a measurement of the web. The direction it points is still useful: compliant crawlers mostly do what robots.txt says. Which is why an edge that contradicts robots.txt is the more interesting failure to hunt.

Scan your site to see the comparison run for you: your home page fetched as a browser and then as four real crawler user agents, with the status code, response size and challenge detection for each, so you can tell an edge block from a robots.txt disallow without reading a single log line. Per-agent detail lives in the crawler registry, and the exact probe procedure is on the methodology page.