Blocklist · OpenAI

Sites that block OAI-SearchBot

608 of the 29,991 domains we have measured, 2%, disallow oai-searchbot at the site root in robots.txt. OAI-SearchBot builds the index that decides whether the operator's assistant cites a page.

Every domain here has removed itself from OpenAI's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

The measurement

608 of 29,991 measured domains (2%) disallow oai-searchbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block OAI-SearchBot", measured 2026-11-21. https://crawlcensus.com/blocked/oai-searchbot

The list 608 domains

RankDomainTLDAI access score
50,598 dailycal.org .org 74
50,727 namur.be .be 67
50,788 liverpool.com .com 47
50,916 motorcycle.com .com 83
50,933 nwaonline.com .com 73
50,982 nrtool.st .st 54
51,009 treasure-maps.com .com 51
- asteroidsathome.net .net 64
PreviousPage 4 of 4

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the OAI-SearchBot reference, the full census, and the JSON endpoint.