Find out what AI can actually read
on your website.
Answer engines do not see the page your visitors see. They see whatever survives robots.txt, your edge firewall, and a fetch with no JavaScript. Crawl Census measures exactly that, for any domain, in about six seconds.
Free, no account. Reports are public so they can be cited and linked.
What gets blocked, crawler by crawler share of measured sites
Percentage of measured sites whose robots.txt disallows each crawler at the site root. Click a bar for the crawler's page and the full blocklist.
Recent policy changes
- score openai.com changed score (40 → 62) 23 minutes ago
- score openai.com changed score (40 → 62) 23 minutes ago
- score openai.com changed score (40 → 62) 23 minutes ago
- score wikipedia.org changed score (44 → 69) 24 minutes ago
- score nytimes.com changed score (35 → 65) 26 minutes ago
- score stripe.com changed score (58 → 91) 26 minutes ago
- score anthropic.com changed score (52 → 72) 26 minutes ago
- score developer.mozilla.org changed score (49 → 72) 26 minutes ago
Recently scanned
| Domain | Score | Answer engines | Scanned |
|---|---|---|---|
| newrepublic.com | 69 | open | just now |
| schema.org | 80 | open | just now |
| secondlife.com | 67 | open | just now |
| semana.com | 83 | open | just now |
| successfactors.eu | 65 | open | just now |
| up.pt | 60 | open | just now |
| adac.de | 68 | open | just now |
| devline.tv | 44 | open | just now |
| geocaching.com | 62 | open | just now |
| enjoei.com.br | 83 | open | just now |
What the score measures 100 points
Can the crawler get the bytes at all? robots.txt groups for 39 tracked agents, wildcard rules, crawl-delay, meta and header directives, plus live requests sent as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot to catch firewall blocking that robots.txt never mentions.
Is there text in the HTML? Word count and text-to-markup ratio measured with JavaScript disabled, main and article landmarks, heading structure, title and description quality.
Can a machine tell what the page is about? JSON-LD validity and entity coverage, canonical URL, sitemap, and the tables, lists and question headings that answer engines lift verbatim.
Can it credit you? Author and organization entities, sameAs links, dateModified freshness, declared licence, and a conforming llms.txt.
Every check is documented on the methodology page, including the exact thresholds and how each one is measured.
Why this is worth measuring context
Blocking a training crawler and blocking an answer engine are different decisions with different consequences, and most sites make them by accident with a single copied robots.txt block. GPTBot collects text that may train a model. OAI-SearchBot builds the index that decides whether ChatGPT cites you. Disallow both and you have not protected anything, you have removed yourself from the results while your competitor stays in them.
The second failure is quieter. A permissive robots.txt means nothing if the edge returns 403 to any user agent containing "bot", which is the default behaviour of several managed rule sets. That is why every report here includes live requests sent under real crawler user agents, not just a reading of the file.
The third is structural. Most AI fetchers do not execute JavaScript. A single-page app that renders its content client-side ships an empty shell to the crawler no matter how permissive the rules are, and the engine summarises the shell.
Crawl Census re-measures continuously and keeps the history, so the record shows not just who is open today but who changed their mind, and when.