Should you block AI crawlers? A decision framework
Blocking a training crawler costs nothing measurable today; blocking a retrieval crawler costs documented visibility, which is why the decision is per-token rather than yes or no.
The answer, and the asymmetry that decides it
Block training crawlers if you object to your work training a model. Do not block retrieval crawlers unless you have decided you do not want to be cited, because that is exactly what blocking them buys.
The asymmetry is the whole decision, and it is unusually clean:
- Blocking a training crawler costs nothing you can currently measure. No vendor documents that allowing a training crawler improves your visibility, ranking or citation rate in any product. Google states directly that
Google-Extended"does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search" (Google crawler documentation, read 2026-08-21). The training crawler is a one-way transfer. - Blocking a retrieval crawler costs visibility today. OpenAI: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links" (OpenAI crawler documentation, read 2026-08-21). That is a documented, immediate consequence.
Which is why the interesting question is almost never "block or not". It is which token, and the vendors have made that separable. The token split guide has the file.
What publishers actually do
The best machine-readable dataset is the Palewire news-homepages tracker (palewi.re news-homepages), which collects robots.txt from news publishers twice a day and archives every observation with a timestamp. Snapshot observed 2026-08-19/20, sample of 1,156 news publishers:
| Crawler group | Opt-outs | Share |
|---|---|---|
| OpenAI | 528 | 45.7% |
| Common Crawl | 528 | 45.7% |
| Google AI | 471 | 40.7% |
| Any of the three | 577 | 49.9% |
Half of news publishers block at least one. Read the caveats before you generalise: the sample is news only and skews to the US and UK, which is the segment with both the strongest legal exposure and the clearest licensing upside. "OpenAI", "Google AI" and "Common Crawl" are the tracker's own groupings and the underlying tokens are not broken out, so this does not tell you whether those sites blocked GPTBot alone or also the search fetcher. A minority of rows carry 2025 timestamps, meaning some sites have not been re-fetched recently.
For contrast, Known Agents scans the SimilarWeb top 1,000 sites daily and reports much lower rates: Bytespider 23%, omgili 19%, Timpibot 15%, YouBot 15%, ImagesiftBot 14%, PanguBot 12%, cohere-training-data-crawler 12% (Known Agents Agentic Web Index, figures as of 2026-08-21). Every heavily blocked agent in that list is a data scraper, not a retrieval agent. The web at large blocks the trainers and leaves the citation fetchers alone, and does it far less aggressively than news does. Note also that Known Agents' own wording is sites that "currently have robots.txt rules for" a bot, which is not identical to blocking it.
The traffic bargain, and why it is not being honoured
The implicit deal is content for referrals. Cloudflare measured whether that holds by dividing HTML crawl requests by HTML referrals per platform (Cloudflare Radar crawl-to-refer ratio, published 2025-07-01). For 19 to 26 June 2025, the ratios ran from Anthropic at 70,900:1 down to Mistral at 0.1:1 — roughly 71,000 page requests per referral at one end, and ten referrals per crawl request at the other. Week-over-week change ranged from +6% for DuckDuckGo and Yandex to −19.4% for Google, the last tied to a Googlebot crawl drop starting 24 June.
Referer: header, and we believe that the same holds true for traffic generated from other native apps as well. As such, because the referral counts only include traffic from the Web-based tools from these providers, these calculations may overstate the respective ratios, but it is unclear by how much." The 70,900:1 figure is an upper bound on the imbalance, not a measurement of it. Cloudflare also excludes Google referral traffic from AS15169 because prefetching does not represent active consumption, and notes GPTBot had multiple periods of little-to-no crawling during June 2025, so ratios move with crawl patterns as much as with referral behaviour.The second data point points the same way with a different flaw. Known Agents puts AI chat referrals at 0.1% of human visits, down 4% over 90 days, as of 2026-08-21. That figure comes from telemetry on their own customer sites — a self-selected sample of operators who already care enough about bot traffic to buy a product for it, not a random sample of the web, and published by a vendor selling the products these statistics motivate. Treat it as an order of magnitude, not a rate.
Two imperfect measurements, both pointing the same direction: AI referral traffic is currently too small to be the reason you allow anything. If your only argument for allowing crawlers is the traffic they send back, the evidence does not support you. The argument that does survive is presence in the answer itself, which is a brand and consideration outcome rather than a session in your analytics, and which no publicly available dataset measures well.
What blocking does and does not achieve
robots.txt is a request, not a lock. Three limits are worth stating plainly.
- It is not enforcement. Known Agents puts overall robots.txt compliance at 96.4% (2026-08-21, same self-selected sample), which is high and is not 100%. More importantly, several documented agents state in advance that they may ignore it: Perplexity-User "generally ignores robots.txt rules", meta-externalfetcher "may bypass robots.txt rules", ChatGPT-User's "robots.txt rules may not apply". A
Disallowagainst a user-initiated fetcher is a preference, not a control. - It is not retroactive. Blocking a training crawler today does nothing about any model already trained on pages you published earlier, and nothing about copies in third-party corpora. Common Crawl's archive is the obvious case: CCBot content collected in previous years remains in datasets built from it.
- It is not a licence, and this page is not legal advice. If your position needs to be enforceable, the layers that carry weight are your terms of service, and edge enforcement that actually refuses the request. robots.txt is the declaration; the firewall is the mechanism. That distinction is why a Crawl Census report shows the robots.txt verdict and a live probe result separately instead of reconciling them — a permissive file plus a WAF rule that 403s anything with "bot" in the user agent is a common and confusing combination, in both directions.
Licensing as the third option
Blocking a training crawler preserves something that allowing it destroys: the ability to sell access to the same content later. That is the strategic argument behind the news numbers above, and it is why the segment with the most licensing leverage blocks at nearly twice the rate of the general top 1,000.
Two honest limits on that reasoning. First, we have no verified public figures for what these agreements pay, so treat any specific number you encounter as unsourced until you see the source; none is cited here because none was verifiable. Second, leverage is a function of corpus scale and distinctiveness. A large archive of original reporting or proprietary data has something to negotiate with. A 40-page marketing site does not, and blocking a trainer to protect a negotiating position that does not exist is a cost with no matching benefit.
The decision framework
| If you are | Training tokens | Retrieval tokens | Reasoning |
|---|---|---|---|
| A news or research publisher with an archive | deny | allow | Preserve licensing leverage and stay citable. This is what the 49.9% is mostly doing. |
| SaaS, commerce, or any business that wants to be recommended | optional | allow | Being named in the answer is the point. Denying trainers is cheap; denying retrieval is self-harm. |
| A documentation or developer-tools site | allow | allow | Your content becoming the default answer for how to use your product is the outcome you want. This is why the AI labs publish llms.txt for their own docs. |
| A personal site, portfolio, or small blog | preference | allow | There is no revenue at stake either way, so decide on principle. Whatever you decide, do not block retrieval by accident with a blanket rule. |
| Behind a paywall or subscription | deny | allow | Allow retrieval of what is public and mark the rest. Applebot reads page-level isAccessibleForFree; section-level hasPart markup is explicitly unsupported (Applebot documentation), so a publisher who marked up only the paywalled section has not opted that content out. |
Whatever you choose, the failure mode is the same for everyone: a blanket Disallow written years ago against a wildcard agent, or a firewall rule nobody remembers deploying, quietly removing you from answer engines you never decided to leave. That is worth checking rather than assuming.
Run a scan to see which training tokens and which retrieval tokens your robots.txt actually covers, whether your edge refuses real crawler user agents regardless of what the file says, and how your posture compares with the rest of the census.