Know before you fetch
Everything else on this site is written for the people being crawled. This page is for the people doing the crawling. The same measurement that tells a publisher how open their site is tells you, more usefully, which doors are actually shut before you spend a request finding out.
The problem a robots.txt parser cannot solve three states, not two
A crawler usually decides where to go by reading robots.txt. That answers one question and hides two others. 3,059 measured domains publish a robots.txt that permits AI agents and then refuse those same agents at the network edge. A parser sees permission; the fetch returns 403. The budget is spent either way.
In the other direction, 213 domains answer an AI user agent with HTTP 402 Payment Required. That is not a refusal, it is a price. Treating it as a block means walking away from content the operator actively wants to sell you; treating it as an allowance and retrying around it means taking something someone is trying to charge for. Neither is what a robots.txt parser will tell you.
Crawl Census sends a real request as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot from a datacentre address, on a rolling seven-day cadence, and records what actually came back. That measurement is what this endpoint returns.
Preflight one call, up to 1,000 domains
Send the agent token you crawl as and a list of domains. Get back a verdict per domain and a summary you can log.
curl -X POST https://crawlcensus.com/api/v1/preflight \
-H 'content-type: application/json' \
-d '{"agent":"gptbot","domains":["nytimes.com","clarin.com","cloudflare.com"]}'{
"agent": "gptbot",
"summary": { "allow": 1, "disallow": 1, "pay": 1, "refuse": 0, "unknown": 0 },
"results": [
{ "domain": "nytimes.com", "verdict": "disallow",
"reason": "robots.txt disallows gptbot at the site root." },
{ "domain": "clarin.com", "verdict": "pay",
"reason": "Answered HTTP 402 Payment Required to an AI agent." },
{ "domain": "cloudflare.com", "verdict": "allow",
"reason": "robots.txt permits gptbot and a live request carrying its user agent was served." }
]
}allow- robots.txt permits this agent and a live request carrying its user agent was served.
disallow- robots.txt forbids this agent at the site root.
refuse- robots.txt permits it, the edge refused it anyway. The allowance is not real; the fetch will fail.
pay- The origin answered HTTP 402. It will serve this agent on commercial terms.
unknown- Not measured recently enough to answer. Decide for yourself, or submit it for a scan.
25 domains per call anonymously, 200 on Pro, 1,000 on Data. No key needed to try it.
Have the web probed as your agent four slots used, eight is the ceiling
Everything else here is derived from data that already exists. This is the one thing that cannot be: whether a site refuses your specific user agent can only be established by being refused. No robots.txt parser substitutes for it, and no third party publishes it.
The census currently sends live requests as GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot on a rolling seven-day cadence across the whole corpus, and records exactly who refuses each one. Every other tracked agent gets policy data only. If you operate a crawler and want the web measured as it, ask:
Each probed agent is one extra live fetch per domain per scan, so the set is capped at eight and requests are answered by a person rather than granted automatically. Say which token you crawl as and the exact user-agent string; if it is already probed you will be told so and pointed at the data, at no cost.
Once probed, your agent gets its own page, its own blocklist.txt and delta feed, edge-refusal figures in preflight, and inclusion in the public dataset under CC BY 4.0 like everything else here.
Asking is free and a slot may well be granted on the merits, particularly for a crawler this census already reports on. If you would rather not wait on a person, the Crawler plan at 499 dollars a month carries a slot outright, along with preflight at 5,000 domains a call. It is priced against what it costs to run: one probed agent is one extra live request against every domain in the corpus, every seven days.
When we have not measured it a base rate beats nothing
On a real crawl a large share of any batch is simply not in this corpus, and "not measured" is a true answer that helps nobody. Those verdicts now carry what comparable measured domains do.
{"domain":"unmeasured-xyz.de","verdict":"unknown",
"prior":{"basis":".de domains measured in this corpus","sample":617,
"refuse_percent":14.4,"disallow_percent":17.3,
"caveat":"A base rate, not a verdict for this domain."}}The differences are real and worth acting on: .de disallows AI agents in robots.txt at 17.3 percent against 6.8 percent for .com, while .com refuses more often at the edge. A crawler deciding whether to spend the request, or to queue the domain for measurement first, is better off with that than with silence.
It is a base rate and it is labelled as one in every response. It is not this domain's policy, it must not be recorded as such, and the caveat travels inside the object rather than living on this page. Only groups of 200 or more measured domains are used; below that the corpus-wide rate is returned instead.
When the answer is no what else the origin publishes
A refusal is only half an answer. Most domains that block a crawler still publish machine-readable content on purpose - a feed, a sitemap, an llms.txt - so every disallow, refuse and pay verdict now carries what that origin publishes and you are permitted to fetch.
{"domain":"forbes.com","verdict":"disallow",
"reason":"robots.txt disallows gptbot at the site root. It does still publish
machine-readable content you are permitted to fetch; see alternatives.",
"alternatives":{"sitemap":"https://www.forbes.com/news_sitemap.xml"}}Only paths the agent is actually permitted are listed, evaluated against that domain's own robots.txt with the same RFC 9309 matcher. Offering a feed that robots.txt disallows would be worse than offering nothing, and the declared sitemap location is used rather than a guessed /sitemap.xml.
The honest headline is that this rarely helps. Across a 121-domain sample of origins that block or charge GPTBot, only 7 percent left any route open: blocking is overwhelmingly total, because Disallow: / covers the sitemap and the feed as well as the article. The field is populated when it is real and absent when it is not, so its emptiness is itself information - it tells you not to go looking.
If your crawler is not on our list it still gets an answer
This census tracks a fixed registry of agents, and every page here is built around it. That quietly favours the incumbents: a smaller or newer operator would get "unknown agent" and nothing, while the same stored robots.txt sitting in this database could answer them perfectly well.
So it does. Send any crawler token and preflight resolves it against each domain's stored policy file using the same RFC 9309 matcher used for tracked agents.
curl -X POST https://crawlcensus.com/api/v1/preflight -H 'content-type: application/json' -d '{"agent":"my-research-bot","domains":["nytimes.com"]}'
{"agent":"my-research-bot","tracked":false,
"results":[{"domain":"nytimes.com","verdict":"allow",
"reason":"robots.txt permits "my-research-bot" at the site root, via the wildcard group..."}]}The limit travels with the answer rather than being buried here: live user-agent probes run for four agents only, so an untracked token gets a robots.txt verdict and an explicit statement that edge refusal for that exact string has not been measured. Want your crawler probed and given its own page and blocklist file? Open an issue on the registry; it costs nothing and the registry is public.
Or just take the file one request, no integration
If your fetcher only needs a deny list in memory, skip the per-domain calls. One plain-text file per agent, one domain per line, commented header, cacheable.
curl https://crawlcensus.com/agents/gptbot/blocklist.txt# Domains that disallow GPTBot in robots.txt
# agent: gptbot
# generated: 2026-08-23T07:27:08.000Z
# count: 3309
073m.com
...Then stay current without re-downloading it. Every response carries next_since, so a long-running crawler costs a few hundred bytes an hour instead of a few hundred kilobytes a day.
https://crawlcensus.com/agents/gptbot/changes.json?since=1787467651
{"count":75,"next_since":1787470076,"changes":[
{"at":"2026-08-23T02:57:07Z","domain":"cbssports.com","change":"now_blocks_you"},
{"at":"2026-08-23T03:02:13Z","domain":"vinted.co.uk","change":"no_longer_blocks_you"}]}Also available as blocklist.json. These files cover robots.txt only. Edge refusal and HTTP 402 are per-request behaviours, so they still need the preflight check above.
Drop-in client JavaScript and Python, no dependencies
An endpoint only helps if it reaches your pipeline. The client turns the measurement into behaviour: it skips domains that will refuse you, caches verdicts per host, and by default declines to route around an origin that quoted a price.
npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-clientimport { politeFetch } from "crawl-census-client";
const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason); // disallow | refuse | pay
else process(await r.response.text());from crawl_census import partition
p = partition(urls, agent="gptbot")
crawl(p.crawl) # p.skip and p.pay never cost you a requestSkipping is returned rather than raised, because it is the normal outcome for a large share of the web. If this census is unreachable every verdict becomes unknown and your crawl proceeds as it normally would: a third-party outage must not stop your pipeline, and there is a live test for exactly that. Source and tests.
Over MCP for agents rather than pipelines
The same check is exposed as an MCP tool, so a model can ask before it fetches rather than after it fails.
{"mcpServers":{"crawl-census":{"url":"https://crawlcensus.com/mcp"}}}Tools: crawl_preflight, site_report, scan_site, census_stats. No authentication for read tools. Full description at /mcp-info.
Be told, rather than poll one email, only when it changes
Everything above is a pull: you have to come and ask. If you run one of these agents, this is the push. One message when sites start or stop disallowing your token, and silence otherwise.
Double opt-in, one address per agent, unsubscribe by replying. The same information is always available without giving an address at /agents/<agent>/changes.json - the email exists because a change you find out about a week late has already cost you the crawl.
Why we publish this plainly
This project measures how open the web is to AI crawlers, and publishes the result under CC BY 4.0. Handing the same measurement to crawler operators is not a contradiction: a crawler that knows where it is unwelcome stops knocking, and a crawler that can see a price has the option of paying it.
Measured 2026-08-23 across 24,879 domains. Every figure here is reproducible from the open dataset and the published method. If your agent is measured here and something looks wrong, prove control of the domain or email hello@crawlcensus.com and it gets fixed.