Crawl Census
Scan a site
Pricing

The measurement is free.
Watching it change is not.

Every audit, every public report and the whole census stay open. The paid tiers exist because re-scanning a domain every day, keeping its history and telling you the hour something changed costs money to run.

Free
$0 forever

For anyone checking one site, once.

  • •Unlimited public single-page audits
  • •A permanent public report page per domain, citable and linkable
  • •Score badge SVG for your own site
  • •60 API calls per day per IP, no key required
  • •Full access to the census, rankings, change log and /data.json
Run an audit
Pro
$19 / month

For a team that has to notice a policy change the day it happens, not the quarter after.

  • •25 monitored domains, re-scanned every day
  • •Email alert on any crawler policy change
  • •90 days of per-domain history
  • •Competitor comparison across your monitored set
  • •Private reports, excluded from the public census
  • •CSV export of every scan and event
  • •10,000 API calls per month

Founding accounts keep this price permanently, including through later increases.

Data
$199 / month

For researchers and vendors that need the whole corpus, not one domain.

  • •The full corpus as a daily JSON and CSV dump
  • •Per-bot time series, one row per bot per day
  • •250,000 API calls per month
  • •Bulk scan endpoint, many domains per request
  • •Commercial redistribution licence, no attribution requirement

Founding accounts keep this price permanently, including through later increases.

Questions worth asking before you pay faq

What counts as a monitored domain?
One registrable hostname, for example example.com. A monitored domain is re-scanned on a schedule whether or not you visit the site, its result is compared against the previous scan, and any difference becomes an event you can be emailed about. Scanning example.com and blog.example.com uses two of your 25 slots, because they can serve different robots.txt files and different markup. Running a free one-off audit never consumes a slot.
How often do re-scans actually run?
A monitored domain is re-scanned once every 24 hours. A free audit runs when you ask for it and is not repeated. There is no faster tier: crawler policy is edited by hand in a text file, so sub-daily polling would add load to the sites we measure without finding anything a daily pass misses.
Is your scanning polite?
Yes, and the numbers are small enough to state exactly. One scan requests nine URLs on your origin and stops: /, /robots.txt, /llms.txt, /ai.txt, /sitemap.xml, then / four more times under the GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot user agents to see whether your edge treats them differently. No links are followed, no assets are fetched, no JavaScript runs. Every request has a hard timeout between 5 and 10 seconds and a capped body read, so a slow or enormous page cannot hold a connection open. The well-known files are requested as CrawlCensusBot/1.0 with +https://crawlcensus.com/bot in the user agent, so the hit is identifiable in your logs and the contact URL is one click away. Disallowing the CrawlCensusBot token in your own robots.txt opts the domain out of scheduled re-scanning, and mailing hello@crawlcensus.com removes an existing report.
Refunds?
Plans are monthly and cancel from the dashboard, effective at the end of the period already paid for. Mail hello@crawlcensus.com within seven days of a charge and it is refunded in full, no questions and no retention offer.
What can I do with the data?
The public dataset at /data.json and everything on the site is licensed CC BY 4.0. That already permits commercial use, including inside a paid product, provided you carry the attribution string Source: Crawl Census (crawlcensus.com). The Data tier does not unlock the numbers, it removes the attribution requirement and adds a written redistribution licence for cases where per-view attribution is impractical, plus the bulk endpoints and daily dumps that make ingesting the corpus practical.
What is not for sale?
Position in the census. There is no sponsored placement, no paid removal of a low score, and no way to change a result other than changing what your server returns. A private Pro report is withheld from the public census entirely, but a domain cannot be published with an improved score.