Open dataset · CC BY 4.0

The AI crawler census

Every domain here was measured the same way: read its robots.txt, resolve the rules for 48 tracked AI agents, then send live requests as four of them and record what came back. No survey, no self-reporting. Figures update as the crawler works through the queue.

What share of the web can an AI crawler actually read the honest denominator

The headline rate elsewhere on this site, 19%, is measured across the 29,997 websites that served this crawler a page. It leaves out the 6,009 websites that refused before any question about AI could be asked - a bare 403 to an ordinary browser user agent, no robots.txt consulted, no content returned.

Excluding them understates the answer, because a site that refuses every programmatic client refuses AI crawlers most of all. Counted properly, of 36,006 websites in the corpus, 11,623 - 32% - cannot be read by an AI crawler: 6,009 refuse outright and 5,614 of the remainder refuse an AI user agent specifically.

19%
refuse AI agents, of sites that answered
32%
unreadable by an AI crawler, of all websites
6,009
refuse every programmatic client

The same exclusion could have distorted the robots.txt figures, and it does not. Those 6,009 sites refused to serve a page but still served a policy file, because robots.txt is fetched in the same pass rather than after the home page succeeds. Their stated policy is indistinguishable from everyone else's: 12.4% disallow an answer engine against 9.5% among sites that answered, and 17.1% disallow a training crawler against 14.3%. A site's written policy turns out to be independent of whether its infrastructure tolerates a programmatic client at all - which is the same split this census keeps finding between what operators decide and what their stack does by default. The robots.txt rates elsewhere on this site are therefore left on the narrower denominator, because widening it changes nothing.

Both numbers are published because they answer different questions. The narrower one is what this crawler directly measured. The wider one is what a person asking "how much of the web can a model actually read" should be told, and it is nearly twice as large. Among the sites refusing outright are openai.com, chatgpt.com and perplexity.ai, whose own products depend on crawling everyone else.

What the corpus is made of 50,061 hostnames

The seed list ranks hostnames by traffic, and a large share of the most trafficked hostnames on the internet are not websites: CDN endpoints, update services and API hosts that answer enormous numbers of DNS queries and serve no page at their apex. akamaiedge.net, cloudfront.net, windowsupdate.com and aaplimg.com all sit in the first hundred and none of them has a home page.

OutcomeHostnamesShare
Website measured29,99759.9%
No website at this hostname13,00426.0%
Refused before assessment6,00912.0%
Other HTTP error1,0512.1%

Every rate published on this site divides by the first row only, because a hostname with no website cannot have a crawler policy. The distinction was verified rather than assumed: 32 hostnames returning a Cloudflare origin error were fetched again from a residential connection with redirects followed, and none of them served a page there either.

How much of this depends on how prominent a site is the corpus is Tranco's top 36,000

Every figure on this page describes domains drawn from the Tranco popularity ranking, positions 2 to 35,634. It is not a sample of the web, which is mostly small and obscure, and rates measured here should not be read as web-wide. The gradient below is published so the direction and size of that bias is visible rather than merely disclosed.

Popularity bandDomainsRefuse at the edgeBlock an answer enginePublish llms.txtMean score
Top 1,00048014.2%11.7%18.8%72.3
1,000 to 10,0005,14019.6%10.8%15.2%73.4
10,000 and below23,81718.7%9.4%14.4%73.6
Unranked56014.1%3.2%22.7%77.3

The two block types behave differently, and that is the finding. Declaring a block in robots.txt tracks prominence closely: the most visited thousand domains do it at roughly twice the rate of those ranked ten thousand and below. Refusing an AI user agent at the edge does not follow that curve at all - it peaks in the middle of the ranking, among sites large enough to sit behind a managed WAF but not large enough to have anyone tuning its bot rules. The first number is a decision. The second is mostly a default.

The measurement

Across 29,997 domains measured by direct request, 10% disallow at least one answer engine in robots.txt and 14% disallow at least one training crawler. Independently of robots.txt, 19% refuse an AI crawler user agent at the network edge. 15% publish an llms.txt.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "The AI crawler census", measured 2026-11-22. https://crawlcensus.com/census

29,997
domains measured
73.6
mean AI access score
10%
block an answer engine
19%
refuse AI agents at the edge
5%
of training-crawler rules are blocks
1%
of answer-index rules are blocks
2%
of live-retrieval rules are blocks
15%
publish an llms.txt

How the web's posture is moving share of measured sites, by day

% of sites20151050% of sites09-2310-2211-21block an answer engineblock a training crawlerpublish llms.txt

Block rate by crawler the headline table

CrawlerOperatorContent used forBlock rateSites blockingSites measured
GPTBot OpenAI training 9.8% 2,937 29,991
CCBot Common Crawl Foundation archive 9.6% 2,881 29,991
Bytespider ByteDance training 9.4% 2,817 29,991
ClaudeBot Anthropic training 8.6% 2,574 29,991
Meta-ExternalAgent Meta training 7.8% 2,340 29,991
Amazonbot Amazon training 7.7% 2,298 29,991
Applebot-Extended Apple training 7.6% 2,291 29,991
Google-Extended Google training 7.6% 2,269 29,991
anthropic-ai Anthropic training 7.3% 2,193 29,991
omgilibot Webz.io training 7.2% 2,164 29,991
Diffbot Diffbot training 7.1% 2,120 29,991
cohere-ai Cohere live retrieval 7.0% 2,112 29,991
omgili Webz.io training 6.9% 2,055 29,991
Google-CloudVertexBot Google live retrieval 4.5% 1,361 29,991
GoogleOther Google training 4.4% 1,314 29,991
MistralAI-Training Mistral AI training 3.8% 1,130 29,991
SemrushBot-OCOB Semrush training 3.5% 1,059 29,991
PerplexityBot Perplexity answer index 3.4% 1,006 29,991
ChatGPT-User OpenAI live retrieval 3.3% 983 29,991
FacebookBot Meta training 3.1% 919 29,991
YouBot You.com answer index 2.9% 876 29,991
ImagesiftBot ImageSift (Hive) training 2.7% 817 29,991
Timpibot Timpi training 2.7% 811 29,991
Scrapy Zyte (open-source framework) training 2.4% 706 29,991
AI2Bot Allen Institute for AI training 2.3% 692 29,991
cohere-training-data-crawler Cohere training 2.1% 615 29,991
OAI-SearchBot OpenAI answer index 2.0% 608 29,991
Meta-ExternalFetcher Meta live retrieval 1.8% 553 29,991
PanguBot Huawei training 1.7% 502 29,991
Claude-User Anthropic live retrieval 1.7% 498 29,991
DuckAssistBot DuckDuckGo live retrieval 1.7% 498 29,991
Perplexity-User Perplexity live retrieval 1.6% 476 29,991
Claude-SearchBot Anthropic answer index 1.5% 463 29,991
MistralAI-User Mistral AI live retrieval 1.5% 457 29,991
Kangaroo Bot Kangaroo LLM training 1.4% 433 29,991
Meta-WebIndexer Meta answer index 1.3% 402 29,991
Applebot Apple answer index 1.2% 350 29,991
Amzn-SearchBot Amazon answer index 0.8% 253 29,991
TikTokSpider ByteDance live retrieval 0.8% 247 29,991
Amzn-User Amazon live retrieval 0.7% 208 29,991
MistralAI-Index Mistral AI answer index 0.4% 111 29,991
facebookexternalhit Meta live retrieval 0.2% 56 29,991
Meta-ExternalAds Meta training 0.1% 41 29,991
Googlebot-News Google answer index 0.1% 39 29,991
Bingbot Microsoft answer index 0.1% 35 29,991
msnbot Microsoft answer index 0.1% 18 29,991
Googlebot Google answer index 0.1% 17 29,991
OAI-AdsBot OpenAI live retrieval 0.0% 13 29,991

Block rate is the share of measured sites whose robots.txt disallows that agent at the site root. A site with no robots.txt counts as allowing everything, which is what the standard specifies.

Why this table's total is lower than 29,997. The headline counts every domain that answered over HTTPS. This table counts only domains that already have a stored policy record for that specific agent, which is a snapshot taken at the last daily rollup and therefore trails the live crawl. The two converge as the queue drains. Both numbers are shown rather than reconciled silently.

Grade distribution

29,997Grade B: 10.41k (34.7%)Grade A: 7.55k (25.2%)Grade C: 7.15k (23.8%)Grade D: 3.33k (11.1%)Grade A+: 1.19k (4%)Grade F: 369 (1.2%)29,997sites

Blocking by top-level domain

TLDSitesBlock an answer enginePublish llms.txt
.com14,60310%18%
.org1,8409%10%
.net1,3069%11%
.ru1,1573%9%
.de82317%7%
.io6544%30%
.edu4713%7%
.uk42917%10%
.jp4019%6%
.br3608%14%
.fr32017%5%
.in3026%11%

A TLD is a weak proxy for jurisdiction, since most are open to anyone, but the spread between them is large and consistent. Full segmentation.

Take the data free, attributed

The corpus is published under CC BY 4.0. Attribute as Source: Crawl Census (crawlcensus.com).

Corpus snapshot (JSON) Census totals Crawler registry Change stream Methodology