Blocklist · OpenAI

Sites that block OAI-SearchBot

608 of the 29,991 domains we have measured, 2%, disallow oai-searchbot at the site root in robots.txt. OAI-SearchBot builds the index that decides whether the operator's assistant cites a page.

Every domain here has removed itself from OpenAI's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

The measurement

608 of 29,991 measured domains (2%) disallow oai-searchbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block OAI-SearchBot", measured 2026-11-21. https://crawlcensus.com/blocked/oai-searchbot

The list 608 domains

RankDomainTLDAI access score
31,023 phoenixnewtimes.com .com 71
31,032 howlongtobeat.com .com 49
31,138 ideal.es .es 82
31,176 edp24.co.uk .uk 79
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,352 sportsbusinessjournal.com .com 53
31,381 coventrytelegraph.net .net 47
31,443 saarbruecker-zeitung.de .de 82
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,786 thespec.com .com 64
31,796 winespectator.com .com 82
31,850 avsforum.com .com 43
31,857 weawow.com .com 65
31,871 diariodesevilla.es .es 84
32,151 devonlive.com .com 47
32,267 theargus.co.uk .uk 79
32,388 rgj.com .com 73
32,389 sepe.es .es 73
32,437 hoopshype.com .com 70
32,579 dallasobserver.com .com 77
32,736 chittorgarh.com .com 64
32,771 annualreports.com .com 46
32,803 oxfordmail.co.uk .uk 79
32,806 tvnet.lv .lv 79
32,918 laverdad.es .es 82
33,063 dailypost.co.uk .uk 47
33,254 aan.com .com 74
33,325 hokkaido-np.co.jp .jp 78
33,569 nottinghampost.com .com 47
33,757 dealabs.com .com 78
33,793 observer.co.uk .uk 74
33,931 lancasteronline.com .com 79
34,092 portafolio.co .co 89
34,309 unionleader.com .com 63
34,445 monoskop.org .org 43
34,449 mylondon.news .news 47
34,491 sfweekly.com .com 74
34,737 vibe.com .com 80
34,750 mailplus.co.uk .uk 44
34,822 milanofinanza.it .it 82
34,847 sydsvenskan.se .se 52
34,961 volksfreund.de .de 83
34,973 naplesnews.com .com 73
35,058 gaoding.com .com 62
35,062 dailypress.com .com 79
35,543 news-journalonline.com .com 73
35,760 montgomerycountymd.gov .gov 65
35,800 wn.de .de 80
35,828 dotdashmeredith.com .com 64
35,953 redflagdeals.com .com 44
36,120 elcomercio.es .es 82
36,334 ytmnd.com .com 41
36,356 elgaronline.com .com 45
36,369 gymmasteronline.com .com 63
36,539 nd-aktuell.de .de 78
36,575 dailyecho.co.uk .uk 81
36,778 cwmars.org .org 43
37,504 technews.tw .tw 70
37,674 leicestermercury.co.uk .uk 47
37,700 rutracker.net .net 43
37,902 iso-group.com .com 93
38,114 sportico.com .com 64
38,129 11freunde.de .de 58
38,148 news-press.com .com 73
38,327 inforum.com .com 70
38,388 goldderby.com .com 74
38,409 igt.com .com 48
38,482 royalacademy.org.uk .uk 52
38,507 bournemouthecho.co.uk .uk 79
38,668 stimme.de .de 72
38,814 mas.to .to 31
38,876 elnortedecastilla.es .es 82
39,210 thecentersquare.com .com 78
39,303 watoday.com.au .au 75
39,349 hoy.es .es 82
39,350 thehighline.org .org 77
39,369 santafenewmexican.com .com 76
39,393 zakzak.co.jp .jp 72
39,471 vwvortex.com .com 43
39,531 hankooki.com .com 72
39,536 buscape.com.br .br 80
39,587 hulldailymail.co.uk .uk 47
39,706 indiehackers.com .com 54
39,806 mercateo.com .com 48
39,824 werstreamt.es .es 59
40,013 belfastlive.co.uk .uk 47
40,068 intelligentgolf.co.uk .uk 66
40,317 iza.ne.jp .jp 79
40,339 kompas.tv .tv 76
40,360 noen.at .at 79
40,512 thetelegraphandargus.co.uk .uk 78
40,638 main-echo.de .de 64
40,713 dwds.de .de 64
40,796 elpasotimes.com .com 73
40,874 grid.id .id 64
40,968 kontan.co.id .id 77
41,088 ispe.org .org 67
41,104 stokesentinel.co.uk .uk 47
41,121 lancashiretelegraph.co.uk .uk 78
41,140 rixos.com .com 69
41,204 elpuntavui.cat .cat 36
41,295 newson6.com .com 82
41,519 greenme.it .it 82
41,647 cjonline.com .com 73
41,713 roanoke.com .com 80
41,721 canarias7.es .es 83
41,736 eadt.co.uk .uk 79
41,760 dailybreeze.com .com 76
41,826 defcon.org .org 66
41,958 theledger.com .com 74
41,962 okinawatimes.co.jp .jp 79
42,076 iex.nl .nl 48
42,112 news9.com .com 85
42,113 glasgowtimes.co.uk .uk 79
42,160 tcpalm.com .com 73
42,260 rcgroups.com .com 41
42,262 advrider.com .com 41
42,293 tfd.com .com 69
42,614 citizen-times.com .com 74
42,698 sol.no .no 74
42,862 horseracingnation.com .com 72
42,919 tcrf.net .net 29
43,302 seacoastonline.com .com 69
43,437 derbytelegraph.co.uk .uk 47
43,480 remoteok.com .com 76
43,666 tasty.co .co 79
43,909 gipuzkoa.eus .eus 70
43,983 teambelgium.net .net 59
44,026 gainesville.com .com 66
44,041 cambridge-news.co.uk .uk 47
44,116 eevblog.com .com 76
44,228 notionpress.com .com 64
44,723 dailygazette.com .com 80
44,807 gofeminin.de .de 74
44,874 news-leader.com .com 62
44,878 rtvnoord.nl .nl 73
44,946 cornwalllive.com .com 47
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,158 autoguide.com .com 87
45,159 lansingstatejournal.com .com 62
45,180 pivotalweather.com .com 25
45,272 rtvutrecht.nl .nl 73
45,316 eumetsat.int .int 80
45,356 pokewiki.de .de 55
45,390 tupalo.com .com 57
45,536 bvb-alyans.ru .ru 80
45,571 wz.de .de 76
45,578 chugoku-np.co.jp .jp 80
45,604 beaconjournal.com .com 66
45,680 greenvilleonline.com .com 62
45,704 diariodeleon.es .es 91
45,771 larioja.com .com 83
45,846 amazon.jp .jp 40
45,850 gloucestershirelive.co.uk .uk 47
46,164 gazettelive.co.uk .uk 47
46,168 bizapedia.com .com 60
46,231 registerguard.com .com 66
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,490 djmag.com .com 65
46,635 wikihow.life .life 76
46,765 plymouthherald.co.uk .uk 47
46,805 omroepwest.nl .nl 73
46,820 kanaloco.jp .jp 76
46,836 unibz.it .it 72
46,918 examinerlive.co.uk .uk 47
46,923 alisweb.org .org 39
46,990 fixya.com .com 43
47,029 lostarmour.info .info 51
47,070 posta.com.tr .tr 82
47,173 eurogamer.de .de 57
47,299 rtvdrenthe.nl .nl 73
47,595 colnect.com .com 63
47,796 topekalibraries.info .info 46
47,884 vcstar.com .com 62
47,916 transphoto.org .org 54
47,948 theafricareport.com .com 55
48,046 trinidadexpress.com .com 74
48,404 lesoleil.com .com 69
48,544 darkhorse.com .com 77
48,550 argusleader.com .com 62
48,859 castro.fm .fm 67
49,059 citypopulation.de .de 54
49,061 posfie.com .com 73
49,124 infolibre.es .es 86
49,194 mailonsunday.co.uk .uk 61
49,279 presstelegram.com .com 80
49,343 hd.se .se 51
49,420 gds.it .it 77
49,501 sj-r.com .com 64
49,574 montgomeryadvertiser.com .com 62
49,611 gamedesire.com .com 85
49,925 aamulehti.fi .fi 66
50,076 dublinlive.ie .ie 47
50,278 burlingtonfreepress.com .com 62
50,312 mujerhoy.com .com 80
50,367 blipfoto.com .com 57
PreviousPage 3 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the OAI-SearchBot reference, the full census, and the JSON endpoint.