Crawl Census
Scan a site
Pricing

All of it is free.
There is nothing to buy.

This page used to sell three subscriptions. They are withdrawn, and the reason is written below rather than quietly deleted, because a census that publishes other people's measurements should publish its own.

What you can do without paying or signing up the whole offer

Measuring
  • Audit any domain, as often as the rate limit allows: 40 live scans an hour per address.
  • A permanent public report page per domain, citable and linkable.
  • A score badge SVG for your own site.
  • Follow one domain per email address and get an alert when its crawler policy changes.
  • Verify a domain you control to follow it on top of that, get a daily re-scan, and ask for removal from the public index.
Reading the data
  • 240 API calls an hour per address, no key.
  • Crawl preflight for 25 domains a call, and the MCP tool over streamable HTTP.
  • The whole corpus at /data.json, cursor-paged so a copy never skips or repeats a row.
  • Per-agent blocklists, wasted-request lists and a cursor-resumable change feed.
  • The census, rankings, segments and change log, and citable facts with a reproduction recipe for each.

Everything is licensed CC BY 4.0. Commercial use is permitted, including inside a paid product, as long as you carry the attribution string Source: Crawl Census (crawlcensus.com).

Why this is not a paid product three experiments, no buyer

Three business experiments were run against the paid hypothesis and all three failed, so the subscriptions were withdrawn rather than left up to collect nothing. No external user existed to sell to: zero settled payments, zero external API keys, zero monitored domains, zero completed domain claims, zero enquiries.

Both plausible buyers already hold better first-party data than this can sell them. A site operator has their own server logs; a crawler operator has their own fetch results. The one asset neither could reproduce is the cross-domain view over time, and Cloudflare Radar publishes that free, with an API, from real traffic across 330 cities.

The full record, including the arithmetic and what it cost, is at /postmortem.

If a paid tier ever reopens it will be because someone asked for it and named a price. There is no order form and no checkout. You can still say you would have paid, and that is recorded and read — it is the one signal that would reopen the question — but it buys nothing and implies no timeline.

Questions about how it works faq

What counts as a followed domain?
One registrable hostname, for example example.com. A followed domain is re-scanned on a schedule whether or not you visit the site, its result is compared against the previous scan, and any difference becomes an event you can be emailed about. Following example.com and blog.example.com counts as two, because they can serve different robots.txt files and different markup. A one-off audit never counts.
How often do re-scans actually run?
A followed domain is re-scanned once every 24 hours, and a verified one is prioritised after you push a fix. A one-off audit runs when you ask for it and is not repeated. Crawler policy is edited by hand in a text file, so polling faster than daily would add load to the sites being measured without finding anything a daily pass misses.
Is your scanning polite?
Yes, and the numbers are small enough to state exactly. One scan requests nine URLs on your origin and stops: /, /robots.txt, /llms.txt, /ai.txt, /sitemap.xml, then / four more times under the GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot user agents to see whether your edge treats them differently. No links are followed, no assets are fetched, no JavaScript runs. Every request has a hard timeout between 5 and 10 seconds and a capped body read, so a slow or enormous page cannot hold a connection open. The well-known files are requested as CrawlCensusBot/1.0 with +https://crawlcensus.com/bot in the user agent, so the hit is identifiable in your logs.
What can I do with the data?
Everything on the site and the dump at /data.json is CC BY 4.0, which permits commercial use inside a paid product provided you carry Source: Crawl Census (crawlcensus.com). There is no tier that removes the attribution requirement, because there are no tiers.
What is not for sale?
Position in the census, and that was true when there were prices too. There is no sponsored placement, no paid removal of a low score, and no way to change a result other than changing what your server returns. A verified owner can ask for their domain to be withheld from the public index entirely; a domain cannot be published with an improved score.
Something here is wrong about my site. Who do I tell?
Mail hello@crawlcensus.com. Every published figure carries a reproduction recipe at /api/v1/facts, so a disagreement can be demonstrated rather than argued about, and a correction that holds up is applied to the corpus rather than to one page.