An independent record
of who the web lets in.

Crawl Census measures one thing: what an AI crawler or answer engine actually receives when it requests a page. It reads robots.txt against 39 tracked crawler tokens, sends live requests under four real crawler user agents to catch blocking that robots.txt never mentions, scores the HTML the way a fetcher without JavaScript sees it, and publishes the result at a permanent URL. Every measured domain is kept, so the record shows not only who is open today but who changed their mind and when.

Why an outside party has to do this

Almost every published statistic about AI crawling comes from a company that is either operating the crawlers or selling the tool that blocks them. Both are legitimate businesses and neither is a disinterested source, because in each case the measured party is also the one publishing the measurement. The check on that is an independent record produced by a method anyone can read and reproduce against their own server logs.

So the method is written out in full, including the parts where it is wrong, the raw dataset is downloadable, and every report is public by default so it can be cited, contradicted, and checked against the site it describes.

How it is funded

Subscriptions, from the Pro and Data tiers. That is the whole model. There is no sponsored placement, no advertising, no affiliate relationship with any crawler operator or bot-management vendor, and no way to pay to change a score. The only thing that moves a result is what your server returns to the next scan. A private Pro report is withheld from the public census in full, but a domain cannot be published with a better number than it earned.

Corrections

Automated measurement produces wrong answers. Re-scanning a domain is free and replaces the stored result immediately, which fixes most of them. For anything a re-scan reproduces, mail hello@crawlcensus.com with the domain and what you expected. Methodology errors are fixed corpus-wide, not just for whoever reported them, and a domain owner can have a report removed from the public census on request without giving a reason. The full policy is on the methodology page.

Licence

Measurements, aggregates and the dataset are published under CC BY 4.0. Republish them, chart them, build on them commercially; just carry the attribution Source: Crawl Census (crawlcensus.com). Crawler names and trademarks belong to their operators and are used descriptively. Nothing here is endorsed by any of them.