Blocklist · OpenAI

Sites that block OAI-SearchBot

608 of the 29,991 domains we have measured, 2%, disallow oai-searchbot at the site root in robots.txt. OAI-SearchBot builds the index that decides whether the operator's assistant cites a page.

Every domain here has removed itself from OpenAI's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

The measurement

608 of 29,991 measured domains (2%) disallow oai-searchbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block OAI-SearchBot", measured 2026-11-21. https://crawlcensus.com/blocked/oai-searchbot

The list 608 domains

RankDomainTLDAI access score
26 amazon.com .com 40
54 tiktok.com .com 49
62 msn.com .com 41
190 cnn.com .com 50
207 bbc.com .com 76
226 bbc.co.uk .uk 77
260 launchpad.net .net 52
318 amazon.co.uk .uk 40
377 amazon.de .de 40
457 cnbc.com .com 54
494 amazon.co.jp .jp 40
609 amazon.fr .fr 40
623 amazon.co.za .za 40
627 dailymail.co.uk .uk 61
635 amazon.ca .ca 40
659 amazon.in .in 40
662 cnet.com .com 79
681 dailymail.com .com 61
706 amazon.it .it 40
725 amazon.es .es 40
812 amazon.com.au .au 40
823 nbcnews.com .com 59
833 amazon.com.br .br 40
834 lefigaro.fr .fr 71
955 corriere.it .it 63
983 elmundo.es .es 84
1,019 theconversation.com .com 62
1,027 huffpost.com .com 85
1,032 amazon.com.mx .mx 40
1,042 spiegel.de .de 64
1,056 marca.com .com 85
1,073 bfmtv.com .com 81
1,098 franceinfo.fr .fr 60
1,177 pimpbunny.com .com 56
1,223 furaffinity.net .net 39
1,275 huffingtonpost.com .com 85
1,286 actu.fr .fr 81
1,300 f95zone.to .to 59
1,305 dw.com .com 42
1,371 sex-studentki.team .team 57
1,376 dagbladet.no .no 74
1,394 usnews.com .com 83
1,409 iltalehti.fi .fi 58
1,446 ekstrabladet.dk .dk 67
1,456 eenadu.net .net 67
1,634 sex-studentki.cafe .cafe 56
1,831 zdnet.com .com 76
1,835 wikihow.com .com 76
1,856 scientificamerican.com .com 82
1,984 themoviedb.org .org 74
1,993 ign.com .com 62
2,006 amazon.eg .eg 40
2,010 mashable.com .com 61
2,115 mirror.co.uk .uk 47
2,118 variety.com .com 80
2,325 nrk.no .no 69
2,423 pcmag.com .com 63
2,433 express.co.uk .uk 48
2,508 yr.no .no 64
2,530 smh.com.au .au 82
2,561 technologyreview.com .com 82
2,581 hollywoodreporter.com .com 79
2,691 newscientist.com .com 76
2,786 chicagotribune.com .com 79
2,917 faz.net .net 39
2,969 today.com .com 62
2,974 thehindu.com .com 66
3,109 ctvnews.ca .ca 82
3,198 metro.co.uk .uk 59
3,253 billboard.com .com 78
3,286 flipboard.com .com 64
3,342 lifehacker.com .com 65
3,368 nzherald.co.nz .nz 56
3,418 deadline.com .com 81
3,433 thefreedictionary.com .com 71
3,499 amazon.se .se 40
3,520 amazon.ae .ae 40
3,576 abc.es .es 81
3,595 politico.eu .eu 62
3,598 amazon.nl .nl 40
3,916 news24.com .com 56
3,952 gitea.com .com 69
4,070 ilsole24ore.com .com 78
4,161 kompas.com .com 77
4,512 amazon.com.tr .tr 40
4,551 yna.co.kr .kr 87
4,553 20minutes.fr .fr 71
4,684 msnbc.com .com 75
4,817 hurriyet.com.tr .tr 81
4,866 channel4.com .com 56
4,987 dr.dk .dk 81
5,057 deccanherald.com .com 64
5,096 nine.com.au .au 71
5,146 everydayhealth.com .com 60
5,153 theage.com.au .au 75
5,158 mercurynews.com .com 78
5,205 lapresse.ca .ca 65
5,229 eltiempo.com .com 85
5,297 chinatimes.com .com 64
5,413 expressen.se .se 61
5,457 diigo.com .com 35
5,547 amazon.ie .ie 40
5,678 dn.se .se 60
5,714 francetvinfo.fr .fr 53
5,752 sankei.com .com 76
5,775 tv2.no .no 70
5,846 runescape.wiki .wiki 81
5,874 hs.fi .fi 68
5,960 milliyet.com.tr .tr 69
6,042 sporcle.com .com 87
6,062 nhk.jp .jp 64
6,089 azcentral.com .com 74
6,125 denverpost.com .com 78
6,159 instiz.net .net 58
6,246 eurogamer.net .net 57
6,308 wwd.com .com 78
6,352 tv2.dk .dk 74
6,356 freep.com .com 73
6,361 ici.fr .fr 87
6,386 huffingtonpost.co.uk .uk 82
6,391 sex-studentki.live .live 56
6,512 telegram.com .com 73
6,513 sandiegouniontribune.com .com 78
6,544 rtvslo.si .si 65
6,610 is.fi .fi 68
6,726 rtbf.be .be 67
6,737 altavista.com .com 53
6,803 schwaebische.de .de 70
6,808 chunichi.co.jp .jp 71
6,816 maxroll.gg .gg 59
6,822 elcorreo.com .com 80
6,895 togetter.com .com 73
6,948 nicematin.com .com 77
6,988 buzzfeednews.com .com 79
7,013 ntv.com.tr .tr 84
7,106 nrc.nl .nl 84
7,119 rp-online.de .de 79
7,127 hani.co.kr .kr 67
7,178 sanspo.com .com 77
7,247 nishinippon.co.jp .jp 69
7,250 manchestereveningnews.co.uk .uk 47
7,268 nzz.ch .ch 76
7,275 baltimoresun.com .com 76
7,324 nordkurier.de .de 69
7,360 mathrubhumi.com .com 82
7,363 bhaskar.com .com 75
7,423 jiji.com .com 81
7,455 amazon.sg .sg 40
7,469 tsn.ca .ca 82
7,510 indiewire.com .com 75
7,531 theonion.com .com 84
7,540 freeones.com .com 71
7,562 nettavisen.no .no 63
7,647 sporza.be .be 87
7,745 huffingtonpost.jp .jp 78
7,863 refinery29.com .com 78
7,937 calciomercato.com .com 81
7,981 afr.com .com 72
8,032 metacritic.com .com 66
8,094 nefisyemektarifleri.com .com 76
8,173 leggo.it .it 62
8,227 sex-studentki.sale .sale 57
8,263 hochi.news .news 78
8,292 walesonline.co.uk .uk 47
8,380 veikkaus.fi .fi 67
8,664 voetbalzone.nl .nl 81
8,799 hmetro.com.my .my 60
8,830 portalsamorzadowy.pl .pl 69
8,854 nikkeibp.co.jp .jp 64
8,883 manhuagui.com .com 61
8,897 rynekzdrowia.pl .pl 64
8,928 nottingham.ac.uk .uk 64
9,062 bluewin.ch .ch 66
9,070 towardsdatascience.com .com 70
9,149 forth.gr .gr 60
9,339 detroitnews.com .com 68
9,388 law.com .com 68
9,542 findnovel.net .net 64
9,925 iconify.design .design 55
10,074 cope.es .es 86
10,279 klingai.com .com 78
10,345 inews.co.uk .uk 67
10,432 expansion.com .com 81
10,470 babycenter.com .com 62
10,501 ocregister.com .com 80
10,621 dailystar.co.uk .uk 47
10,781 47news.jp .jp 67
10,885 livestrong.com .com 80
10,894 augsburger-allgemeine.de .de 81
11,102 letyshops.com .com 60
11,130 lexpress.fr .fr 86
11,154 internshala.com .com 87
11,201 latercera.com .com 80
11,216 nola.com .com 64
11,349 posteo.de .de 67
11,457 wnp.pl .pl 65
11,572 overheid.nl .nl 68
11,642 fap-guru.pro .pro 56
11,751 dailyrecord.co.uk .uk 47
11,914 thehindubusinessline.com .com 65
Page 1 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the OAI-SearchBot reference, the full census, and the JSON endpoint.