Crawl Census
Scan a site
API

Every number on this site,
as JSON.

One versioned REST API over the same store the pages read. No SDK, no GraphQL, no pagination cursors you have to learn. Responses are JSON, CORS is open, and the dataset is CC BY 4.0.

/api/v1
base path
60 / day
unauthenticated limit
Bearer
auth scheme
CC BY 4.0
data licence

Endpoints https://crawlcensus.com/api/v1

GET/api/v1/site/{domain}

The most recent stored audit for one domain.

Returns the cached result without touching the origin. 404 if the domain has never been scanned; POST /api/v1/scan first. Trimmed below: bots carries all 39 registry entries and checks carries all 27.

Request
curl -s https://crawlcensus.com/api/v1/site/example.com
Response
{
  "domain": "example.com",
  "scanned_at": "2026-08-19T04:12:07Z",
  "score": 71,
  "grade": "B",
  "dimensions": { "reach": 26, "read": 22, "structure": 15, "attribute": 8 },
  "blocks_training": 6,
  "blocks_answer": 2,
  "ua_blocked": ["perplexitybot"],
  "has_llms_txt": false,
  "bots": [
    {
      "id": "gptbot",
      "label": "GPTBot",
      "operator": "OpenAI",
      "purpose": "train",
      "allowed": false,
      "probe_status": 200,
      "probe_blocked": false
    },
    {
      "id": "oai-searchbot",
      "label": "OAI-SearchBot",
      "operator": "OpenAI",
      "purpose": "search",
      "allowed": true,
      "probe_status": 200,
      "probe_blocked": false
    },
    {
      "id": "perplexitybot",
      "label": "PerplexityBot",
      "operator": "Perplexity",
      "purpose": "search",
      "allowed": true,
      "probe_status": 403,
      "probe_blocked": true
    }
  ],
  "checks": [
    {
      "id": "robots-blocks-training",
      "state": "fail",
      "earned": 0,
      "weight": 8,
      "detail": "robots.txt disallows 6 training crawlers at /."
    },
    {
      "id": "ua-block-detected",
      "state": "fail",
      "earned": 0,
      "weight": 6,
      "detail": "PerplexityBot received HTTP 403 where a browser user agent received 200."
    },
    {
      "id": "jsonld-missing",
      "state": "pass",
      "earned": 7,
      "weight": 7,
      "detail": "2 JSON-LD blocks parsed: Organization, WebSite."
    }
  ]
}
POST/api/v1/scan

Run a live audit now and return the result.

Fetches the origin, so it is slower than the cached read and it is rate limited harder. The response body is the same shape as GET /api/v1/site/{domain}. A scan of a domain already scanned in the last few minutes returns the stored result instead of re-fetching.

Request
curl -s -X POST https://crawlcensus.com/api/v1/scan \
  -H 'content-type: application/json' \
  -d '{"domain":"example.com"}'
Response (trimmed)
{
  "domain": "example.com",
  "scanned_at": "2026-08-21T09:30:44Z",
  "score": 74,
  "grade": "B",
  "dimensions": { "reach": 32, "read": 22, "structure": 15, "attribute": 5 },
  "blocks_training": 6,
  "blocks_answer": 0,
  "ua_blocked": [],
  "has_llms_txt": false,
  "bots": [
    {
      "id": "gptbot",
      "label": "GPTBot",
      "operator": "OpenAI",
      "purpose": "train",
      "allowed": false,
      "probe_status": 403,
      "probe_blocked": true
    }
  ],
  "checks": [
    {
      "id": "robots-blocks-answer-engines",
      "state": "pass",
      "earned": 12,
      "weight": 12,
      "detail": "No answer-engine fetcher is disallowed at /."
    }
  ]
}
GET/api/v1/census

Aggregate figures across every domain on record.

The same snapshot the home page and /census render. Counts are domains, not pages. Example values.

Request
curl -s https://crawlcensus.com/api/v1/census
Response
{
  "sites": 18432,
  "avg_score": 63.4,
  "blocks_training": 7211,
  "blocks_answer": 2604,
  "has_llms_txt": 913,
  "updated_at": "2026-08-21T04:00:00Z"
}
GET/api/v1/bots

The full crawler registry: 39 AI crawlers and answer-engine fetchers.

Static reference data, safe to cache. token is the robots.txt user-agent token to write rules against; user_agent is the full header the operator sends. respects_robots records what the operator documents, not what we can enforce.

Request
curl -s https://crawlcensus.com/api/v1/bots
Response (trimmed to one entry)
{
  "count": 39,
  "bots": [
    {
      "id": "gptbot",
      "token": "gptbot",
      "label": "GPTBot",
      "operator": "OpenAI",
      "purpose": "train",
      "respects_robots": true,
      "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
      "docs": "https://platform.openai.com/docs/bots"
    }
  ]
}
GET/api/v1/bot/{id}

One registry entry plus how the measured corpus treats it.

block_rate is sites_blocking divided by the number of domains where the answer is known. Example values.

Request
curl -s https://crawlcensus.com/api/v1/bot/gptbot
Response
{
  "id": "gptbot",
  "label": "GPTBot",
  "operator": "OpenAI",
  "purpose": "train",
  "token": "gptbot",
  "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot",
  "respects_robots": true,
  "sites_blocking": 4318,
  "sites_allowing": 14114,
  "block_rate": 0.2343
}
GET/api/v1/changes?since={unix}

Every recorded policy change after a unix-second timestamp.

kind is one of block, unblock, llms-added, llms-removed, score. bot is the registry id for block and unblock events and empty otherwise. old and new are strings so a score change and a rule change share one shape. Poll with the ts of the last event you processed.

Request
curl -s 'https://crawlcensus.com/api/v1/changes?since=1755561600'
Response
{
  "since": 1755561600,
  "events": [
    {
      "ts": 1755563421,
      "domain": "example.com",
      "kind": "block",
      "bot": "gptbot",
      "old": "allow",
      "new": "disallow"
    },
    {
      "ts": 1755567002,
      "domain": "example.org",
      "kind": "llms-added",
      "bot": "",
      "old": "",
      "new": "/llms.txt"
    },
    {
      "ts": 1755571180,
      "domain": "example.net",
      "kind": "score",
      "bot": "",
      "old": "58",
      "new": "74"
    }
  ]
}
GET/data.json

The whole public corpus in one file.

Not under /api/v1 and not rate limited, because it is a single cached object rather than a query. Regenerated daily. The Data tier adds the same snapshot as CSV and the per-bot time series.

Request
curl -s https://crawlcensus.com/data.json
Response (trimmed to one entry per array)
{
  "generated_at": "2026-08-21T04:00:00Z",
  "license": "CC BY 4.0",
  "attribution": "Source: Crawl Census (crawlcensus.com)",
  "census": {
    "sites": 18432,
    "avg_score": 63.4,
    "blocks_training": 7211,
    "blocks_answer": 2604,
    "has_llms_txt": 913
  },
  "bots": [
    { "id": "gptbot", "sites_blocking": 4318, "sites_allowing": 14114, "block_rate": 0.2343 }
  ],
  "sites": [
    { "domain": "example.com", "scanned_at": "2026-08-19T04:12:07Z", "score": 71, "grade": "B", "blocks_training": 6, "blocks_answer": 2, "has_llms_txt": false }
  ]
}

Authentication bearer keys

Sending a key

Pro and Data accounts get an API key. Send it as a bearer token on every request. There is no query-string form, so keys do not end up in logs and referrers.

curl -s https://crawlcensus.com/api/v1/site/example.com \
  -H 'authorization: Bearer cc_live_9f2a7c41d8b34e6f'

A key is scoped to one account. Rotating a key from the dashboard invalidates the previous one immediately.

Without a key

Every read endpoint works unauthenticated at 60 requests per day per IP. That is enough to check a handful of domains or wire up a small dashboard, and it needs no signup. POST /api/v1/scan counts against the same allowance and costs more than one unit, because it fetches a third-party origin.

Pro raises the allowance to 10,000 calls per month and Data to 250,000, counted per account rather than per IP.

Limits, errors and CORS 429

Every response carries the current allowance, whether or not you are near it.

ratelimit-limit: 60
ratelimit-remaining: 41
ratelimit-reset: 1755648000

When the allowance is gone the API returns 429 with a retry-after header in seconds and a JSON body. It never returns a partial or stale result to disguise a limit.

HTTP/2 429
retry-after: 21593
content-type: application/json

{
  "error": "rate_limited",
  "limit": 60,
  "remaining": 0,
  "reset": 1755648000,
  "message": "Unauthenticated limit of 60 requests per day reached. Retry after the reset, or use a key."
}

Other failures use the same envelope with a different error code: 400 invalid_domain for input that is not a resolvable public hostname, 401 invalid_key, 404 not_scanned, and 502 origin_unreachable when a live scan could not reach the site at all. A 502 is a fact about the target, not about this API.

CORS is open. Every endpoint answers with access-control-allow-origin: *, so you can call it from browser JavaScript on any page. There are no credentials in cookies and nothing user-specific in a response, so there is nothing for a cross-origin read to leak.

Licence cc by 4.0

Measurements, aggregates and the dataset are published under Creative Commons Attribution 4.0. Use them commercially, republish them, build on them. One condition: carry the attribution.

Source: Crawl Census (crawlcensus.com)

In a chart, a caption is enough. In an article, a linked sentence is enough. In a product where per-view attribution is impractical, the Data tier includes a written redistribution licence that removes the requirement. Bot names and trademarks belong to their operators and are used descriptively.