# Crawl Census welcomes AI crawlers. That is rather the point.
#
# If you are a crawler, this site is built for you as much as for the sites it measures.
# Before you spend requests finding out which doors are shut, ask:
#
#   curl -X POST https://crawlcensus.com/api/v1/preflight \
#     -H 'content-type: application/json' \
#     -d '{"agent":"YOUR-TOKEN","domains":["example.com"]}'
#
# Verdicts are allow, disallow, refuse (robots permits you, the edge refuses you anyway),
# pay (the origin answers HTTP 402 Payment Required and will serve you on commercial terms) and unknown.
# What each obliges you to do, as data rather than prose: https://crawlcensus.com/api/v1/verdicts
# Works for any token, not only the agents tracked here.
#
#   Deny list, one domain per line:  https://crawlcensus.com/agents/YOUR-TOKEN/blocklist.txt
#   Only what changed since a time:  https://crawlcensus.com/agents/YOUR-TOKEN/changes.json?since=UNIX
#   MCP server:                      https://crawlcensus.com/mcp
#   Permits you then refuses you:    https://crawlcensus.com/agents/YOUR-TOKEN/refused.txt
#   Will sell access to YOU:         https://crawlcensus.com/agents/YOUR-TOKEN/charging.txt
#   Sell to any AI crawler:          https://crawlcensus.com/charging.txt
#   Everything else:                 https://crawlcensus.com/for-crawlers
#
# Want the web measured as YOUR user agent, not only the four probed today? That is the one
# figure no robots.txt parser can give you, and it is asked for at https://crawlcensus.com/for-crawlers
#
# If this census publishes something wrong about your crawler, the correction channel is at
# https://crawlcensus.com/operators. Registry facts are fixed without argument.

User-agent: *
Allow: /
# /scan and /api/v1/scan trigger work on someone else's server. /claim is a verification form,
# linked from every report page, serving the same three hundred words whatever domain is in the
# query string - a fetch per domain in the corpus, returning nothing.
#
# Compliance with these three lines is measured and published at https://crawlcensus.com/compliance. Not as a
# complaint: it is the only robots.txt whose enforcement this census can observe directly rather
# than infer, and a census that publishes what sites ask for should say whether the asking works.
Disallow: /scan
Disallow: /api/v1/scan
Disallow: /claim

Sitemap: https://crawlcensus.com/sitemap.xml