Independent measurement · 29,996 sites on record

Find out what AI can actually read
on your website.

Answer engines do not see the page your visitors see. They see whatever survives robots.txt, your edge firewall, and a fetch with no JavaScript. Crawl Census measures exactly that, for any domain, in about six seconds.

Free, no account. Reports are public so they can be cited and linked.

29,996
domains measured
10%
block at least one answer engine
14%
block at least one training crawler
15%
publish an llms.txt

What gets blocked, crawler by crawler share of measured sites

bar chartGPTBot10%CCBot10%Bytespider9%ClaudeBot9%Meta-ExternalAgent8%Amazonbot8%Applebot-Extended8%Google-Extended8%anthropic-ai7%omgilibot7%Diffbot7%cohere-ai7%omgili7%Google-CloudVertexBot5%

Percentage of measured sites whose robots.txt disallows each crawler at the site root. Click a bar for the crawler's page and the full blocklist.

Recent policy changes

  • edge-serve gm.com stopped refusing at the edge GPTBot 1 day ago
  • edge-serve gm.com stopped refusing at the edge OAI-SearchBot 1 day ago
  • edge-serve gm.com stopped refusing at the edge ClaudeBot 1 day ago
  • edge-serve gm.com stopped refusing at the edge PerplexityBot 1 day ago
  • edge-refuse shopee.co.th started refusing at the edge ClaudeBot 1 day ago
  • unblock amazon.sa unblocked GPTBot 1 day ago
  • unblock amazon.sa unblocked OAI-SearchBot 1 day ago
  • unblock amazon.sa unblocked ChatGPT-User 1 day ago
  • unblock amazon.sa unblocked ClaudeBot 1 day ago
  • unblock amazon.sa unblocked Claude-User 1 day ago

Full change log · RSS

Recently scanned

DomainScoreAnswer enginesScanned
0daymusic.org 52 2 blocked just now
madk-auto.ru 52 open 9 hours ago
xn----7sbarohhk4a0dxb3c.xn--p1ai 77 open 11 hours ago
radialswitchboard.com 71 open 22 hours ago
nios.ac.in 69 open 1 day ago
metasolutions.net 65 open 1 day ago
growthbook.io 96 open 1 day ago
appboy.com 85 open 1 day ago
cyberfolks.pl 98 open 1 day ago
easydns.com 93 open 1 day ago

Full rankings

What the score measures 100 points

Reach · 40

Can the crawler get the bytes at all? robots.txt groups for 39 tracked agents, wildcard rules, crawl-delay, meta and header directives, plus live requests sent as GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot to catch firewall blocking that robots.txt never mentions.

Readability · 25

Is there text in the HTML? Word count and text-to-markup ratio measured with JavaScript disabled, main and article landmarks, heading structure, title and description quality.

Structure · 20

Can a machine tell what the page is about? JSON-LD validity and entity coverage, canonical URL, sitemap, and the tables, lists and question headings that answer engines lift verbatim.

Attribution · 15

Can it credit you? Author and organization entities, sameAs links, dateModified freshness, declared licence, and a conforming llms.txt.

Every check is documented on the methodology page, including the exact thresholds and how each one is measured.

Put it in your pipeline free

GitHub Action

Fail the build when a deploy makes your site less readable to AI. Runs the same audit, writes a job summary, annotates the failing checks.

- uses: taylorsmithgg/ai-access-check@v1
  with:
    domain: example.com
    min-score: 70
Repository
MCP server

Let an agent scan a site and query the census directly over streamable HTTP.

claude mcp add --transport http   crawl-census https://crawlcensus.com/mcp
Tools and schemas
JSON API

Every measurement this site publishes is available as JSON, CORS open, CC BY 4.0.

curl https://crawlcensus.com/api/v1/site/example.com
Endpoints

Why this is worth measuring context

Blocking a training crawler and blocking an answer engine are different decisions with different consequences, and most sites make them by accident with a single copied robots.txt block. GPTBot collects text that may train a model. OAI-SearchBot builds the index that decides whether ChatGPT cites you. Disallow both and you have not protected anything, you have removed yourself from the results while your competitor stays in them.

The second failure is quieter. A permissive robots.txt means nothing if the edge returns 403 to any user agent containing "bot", which is the default behaviour of several managed rule sets. That is why every report here includes live requests sent under real crawler user agents, not just a reading of the file.

The third is structural. Most AI fetchers do not execute JavaScript. A single-page app that renders its content client-side ships an empty shell to the crawler no matter how permissive the rules are, and the engine summarises the shell.

Crawl Census re-measures continuously and keeps the history, so the record shows not just who is open today but who changed their mind, and when.