Sites that charge AI crawlers

The most interesting response is not a single status code but a pair of them. 47 domains refuse the training crawlers outright and quote a price to the retrieval crawlers, in the same second, from the same address.

A request as GPTBot or ClaudeBot returns 403. A request as OAI-SearchBot or PerplexityBot returns 402. The operator is stating a commercial position with precision: do not use this to train a model, and you may cite it in an answer if you pay. That distinction is the substance of every publisher negotiation currently running, and here it is implemented in production and measurable from outside.

No robots.txt file can express this. A policy file can allow or disallow a token; it cannot quote a rate, and it cannot make the answer depend on what the crawler intends to do with the page. This is why a robots.txt-only study cannot see the split at all.

A 402 is only useful to a crawler if it says what it wants. Every response from these domains was captured and searched for the headers a pay-per-crawl scheme would use to state terms - crawler-price, www-authenticate, link, signature-agent and others.

242 of 252 domains answering HTTP 402 state nothing a machine can act on. The 10 that send anything at all send retry-after: 0, which tells a crawler to try again immediately and says nothing about money.

So the status code is being used as a signal rather than as a transaction. An operator publishing 402 is saying "not for free" to an audience that has no way to pay, and the crawler on the other side has no machine-readable path from the refusal to the content. That gap is worth naming precisely, because it is the difference between a working market and a locked door with a price tag nobody can read. Whatever terms exist are arranged out of band, between companies, which is exactly what the status code was meant to replace.

Crawl Census reports this rather than guessing at it: the preflight endpoint returns any captured terms verbatim on a pay verdict, and says plainly when there are none.

A 403 says the operator does not want this crawler to read the page. A 402 says the operator will serve it, on terms. Those are opposite commercial positions and they produce identical rows in any study that buckets by "non-200 response", which is how most crawler-blocking figures are produced.

This distinction is only visible to a measurement that sends a real request. It cannot be derived from robots.txt, because a site quoting a price typically keeps a permissive robots.txt: the transaction happens at the edge, not in a policy file. Of the domains listed here, the majority publish robots.txt rules that allow the very agents they charge.

Status 402 was reserved for this purpose in the original HTTP specification and went essentially unused for thirty years. Its appearance in production on prominent publishers is a measurable change in how the web is monetised, not a configuration error.