The AI crawler census

The headline rate elsewhere on this site, 19%, is measured across the 30,185 websites that served this crawler a page. It leaves out the 5,739 websites that refused before any question about AI could be asked - a bare 403 to an ordinary browser user agent, no robots.txt consulted, no content returned.

Excluding them understates the answer, because a site that refuses every programmatic client refuses AI crawlers most of all. Counted properly, of 35,924 websites in the corpus, 11,560 - 32% - cannot be read by an AI crawler: 5,739 refuse outright and 5,821 of the remainder refuse an AI user agent specifically.

The same exclusion could have distorted the robots.txt figures, and it does not. Those 5,739 sites refused to serve a page but still served a policy file, because robots.txt is fetched in the same pass rather than after the home page succeeds. Their stated policy is indistinguishable from everyone else's: 6.7% disallow an answer engine against 6.6% among sites that answered, and 11.4% disallow a training crawler against 11.7%. A site's written policy turns out to be independent of whether its infrastructure tolerates a programmatic client at all - which is the same split this census keeps finding between what operators decide and what their stack does by default. The robots.txt rates elsewhere on this site are therefore left on the narrower denominator, because widening it changes nothing.

Both numbers are published because they answer different questions. The narrower one is what this crawler directly measured. The wider one is what a person asking "how much of the web can a model actually read" should be told, and it is nearly twice as large. Among the sites refusing outright are openai.com, chatgpt.com and perplexity.ai, whose own products depend on crawling everyone else.

The seed list ranks hostnames by traffic, and a large share of the most trafficked hostnames on the internet are not websites: CDN endpoints, update services and API hosts that answer enormous numbers of DNS queries and serve no page at their apex. akamaiedge.net, cloudfront.net, windowsupdate.com and aaplimg.com all sit in the first hundred and none of them has a home page.

Every rate published on this site divides by the first row only, because a hostname with no website cannot have a crawler policy. The distinction was verified rather than assumed: 32 hostnames returning a Cloudflare origin error were fetched again from a residential connection with redirects followed, and none of them served a page there either.

Every figure on this page describes domains drawn from the Tranco popularity ranking, positions 2 to 35,634. It is not a sample of the web, which is mostly small and obscure, and rates measured here should not be read as web-wide. The gradient below is published so the direction and size of that bias is visible rather than merely disclosed.

The two block types behave differently, and that is the finding. Declaring a block in robots.txt tracks prominence closely: the most visited thousand domains do it at roughly twice the rate of those ranked ten thousand and below. Refusing an AI user agent at the edge does not follow that curve at all - it peaks in the middle of the ranking, among sites large enough to sit behind a managed WAF but not large enough to have anyone tuning its bot rules. The first number is a decision. The second is mostly a default.

The corpus is published under CC BY 4.0. Attribute as Source: Crawl Census (crawlcensus.com).