What one scan does,
request by request.

Five well-known URLs are requested in parallel, then the home page is requested four more times, once under each probed crawler's user agent. Nine requests, one origin, no crawl. No links are followed and no subresources are fetched.

The baseline user agent is a current desktop Chrome string, because the question the report answers is what a crawler receives compared with what a person receives. Measuring both with a bot user agent would hide exactly the discrimination we are looking for.

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36

The well-known files are requested under our own identity, with a contact URL, so the hit is attributable in your logs.

Mozilla/5.0 (compatible; CrawlCensusBot/1.0; +https://crawlcensus.com/bot)

Bodies are read up to the cap and then the read stops; a truncated body is scored as truncated rather than as an error. Every request is sent with caching disabled at our edge, so a report reflects the origin at scan time and not a copy from hours earlier.

The parser follows RFC 9309, which is the specification the major crawler operators say they implement. The rules that decide a verdict, in the order they apply:

1. Group selection by longest matching agent token. Consecutive User-agent lines form one group with shared rules. For a given crawler the group whose token is the longest case-insensitive prefix match of the crawler's product token wins. Googlebot-News is governed by a googlebot-news group even when a googlebot group also exists.
2. Wildcard fallback. With no explicit token match, the * group applies. With no * group either, the crawler is unrestricted, and the report records that the outcome came from a default rather than from a rule.
3. Longest path wins, allow breaks ties. Inside the selected group every rule is matched against the path and the rule with the longest pattern decides. When an Allow and a Disallow match with equal length, Allow wins. Rules in other groups are never consulted, which is why adding a permissive * group does not soften a specific Disallow.
4. Wildcards. * matches any sequence of characters and a trailing $ anchors the pattern to the end of the path, so Disallow: /*.pdf$ blocks /a/b.pdf and not /a/b.pdf?x=1. Patterns with more than 20 wildcards degrade to a literal prefix match rather than compiling a pathological expression.
5. Edge cases that trip people up. An empty Disallow: is a valid line that matches nothing, so it permits everything in its group. %2F is normalized to one case before comparison so encoded slashes compare equal. Unknown or malformed fields do not abort the parse; they are recorded and shown in the report, because a typo in a field name silently disables the rule you thought you wrote.

Verdicts are evaluated at the site root, /. All 39 registry tokens are evaluated against the same document, so a report tells you which crawlers a rule actually covers rather than which ones you meant it to cover.

robots.txt is a request. The edge is the enforcement. A permissive file means nothing if a managed firewall rule returns 403 to any user agent containing "bot", so the home page is requested again under four real crawler user agents and compared against the baseline.

A probe response is classified as blocked when any of the following holds, tested in this order:

• a cf-mitigated response header, which is an explicit statement that a challenge was served
• status 403, 401, 429 or 451
• a 5xx status where the baseline browser request did not produce one, so the failure tracks the user agent and not the server
• a known interstitial body signature: a Cloudflare "just a moment" challenge, an "enable JavaScript and cookies to continue" page, a short "access denied" page, or a CAPTCHA page under 9 KB
• any 4xx where the baseline succeeded

A probe that gets no response at all is recorded as no response, not as a block. The distinction matters: an unreachable origin is not a policy.

Whether AI crawlers and answer-engine fetchers are permitted to request the page at all, in robots.txt, in robots directives, and at the edge.

Whether a fetcher that does not execute JavaScript receives the actual content, in markup an extractor can segment.

Machine-readable markup that states the page's type, entities, canonical URL, and discrete facts instead of leaving them to be inferred.

Signals that let an answer engine name the author, date the content, resolve the publisher, and cite it under known terms.

• One page, not a crawl. Every readability and structure check describes the home page. A site with an excellent home page and client-rendered documentation scores well here and still fails a crawler on the pages that matter. Treat the score as a sample, not a site-wide audit.
• No JavaScript, by design. Nothing is executed. This is not a shortcut, it is the measurement: most AI fetchers do not run scripts either, so the HTML we score is close to the HTML they score. It does mean a hydrated single-page app is judged on its shell, which is exactly the finding.
• robots.txt is a declaration, not enforcement. An Allow proves intent and nothing more. Conversely a Disallow does not prove that anyone obeyed it. The registry records what each operator documents about honoring robots.txt, and several agents document that they do not, or that user-initiated fetches are exempt. Those exemptions are noted per crawler and cannot be verified from outside.
• Two important tokens cannot be probed at all. Google-Extended and Applebot-Extended are robots.txt control tokens for AI training use, not crawlers. They send no user agent of their own, so their entries carry an empty user agent string and are scored from robots.txt only. A live probe is impossible, not merely omitted.
• llms.txt is scored as a deduction, and that is a judgement call. A missing /llms.txt loses points here. The file is a proposed convention with no formal adoption commitment from the major operators, and Chrome Lighthouse treats its absence as not-applicable rather than as a fault. Our position is that a machine-readable index of your own content is cheap and the downside is nothing, so it is scored. Reasonable people disagree; the weight is published above so you can subtract it.
• Point-in-time. A report describes one moment. Rules change, firewalls learn, CDN configurations get deployed on Fridays. The change log exists because the single number is the least interesting part of the record.
• No relationship with any operator. Nothing here is confirmed by OpenAI, Anthropic, Google, Perplexity or anyone else. Registry entries are compiled from each operator's public documentation, linked on every crawler page so you can check them.

Results are automated, so some of them are wrong. The fastest fix is a re-scan, which is free, needs no account, and replaces the stored result immediately: run the domain again from the scan box.

If a re-scan reproduces a result you believe is wrong, mail hello@crawlcensus.com with the domain and what you expected. Useful in the mail: the timestamp of the report, and any edge or WAF rule you think fired. We will re-run it from a different path and either correct the record or tell you what the origin returned and why it was classified the way it was.

When a methodology error is found, the fix is applied to the whole corpus rather than to the domain that complained, and the change is noted on this page. A registry entry that misstates an operator's documented behaviour is corrected on sight. A domain owner can have a report removed from the public census on request, with no requirement to explain.