Ask your assistant
whether AI can read a site.

One endpoint handles the whole protocol. Requests are JSON-RPC 2.0 sent with POST. A response is a JSON object or an SSE stream, depending on the accept header your client sends. There is no stdio build and no separate SSE endpoint to configure.

https://crawlcensus.com/mcp

Unauthenticated calls share the public allowance of 60 requests per day per IP. Sending a Pro or Data key as authorization: Bearer <key> raises it to the account limit. Every tool is read-only with respect to your data: nothing is stored about the client, and a scan_site result is published to the public census the same way a scan from the website is.

Run a live audit of one domain and return the full result.

Fetches the origin, so it takes a few seconds. Rate limited the same way as POST /api/v1/scan.

{
  "name": "scan_site",
  "description": "Audit how AI crawlers and answer engines can read a website. Runs a live scan of the domain and returns its score, per-crawler robots.txt verdicts, live user-agent probe results and every failed check.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "domain": {
        "type": "string",
        "description": "Bare registrable hostname, for example example.com. A full URL is accepted and normalized."
      }
    },
    "required": ["domain"],
    "additionalProperties": false
  }
}

Read the most recent stored audit for a domain without touching its origin.

Returns an error when the domain has never been scanned. Call scan_site first.

{
  "name": "site_report",
  "description": "Return the most recent stored Crawl Census audit for a domain, including score history, without contacting the site.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "domain": {
        "type": "string",
        "description": "Bare registrable hostname, for example example.com."
      },
      "include_checks": {
        "type": "boolean",
        "description": "Include the full 27-check breakdown rather than only the failed checks.",
        "default": false
      }
    },
    "required": ["domain"],
    "additionalProperties": false
  }
}

Aggregate figures across the whole measured corpus.

Takes no required argument. Pass a bot id to narrow the answer to one crawler.

{
  "name": "census_stats",
  "description": "Return aggregate Crawl Census figures: domains measured, average score, how many block training crawlers or answer engines, and how many publish an llms.txt. Optionally narrowed to one crawler.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "bot": {
        "type": "string",
        "description": "Optional crawler registry id, for example gptbot, to return that crawler's block rate instead of the corpus totals."
      }
    },
    "required": [],
    "additionalProperties": false
  }
}

claude mcp add --transport http crawl-census https://crawlcensus.com/mcp

Add --scope user to make it available in every project rather than the current one. Verify with claude mcp list.

Add the server to claude_desktop_config.json and restart the app.

{
  "mcpServers": {
    "crawl-census": {
      "type": "http",
      "url": "https://crawlcensus.com/mcp"
    }
  }
}

Any other MCP client that supports streamable HTTP takes the same URL. Clients that only speak stdio need a local bridge; the HTTP endpoint is the only transport served here.

The protocol is plain JSON-RPC, so a client is not required to test it. List the tools:

curl -s https://crawlcensus.com/mcp \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

And call one:

curl -s https://crawlcensus.com/mcp \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"census_stats","arguments":{"bot":"gptbot"}}}'

If you would rather not speak MCP at all, the REST API exposes the same data over ordinary GET requests.