Blocklist · OpenAI

Sites that block OAI-SearchBot

608 of the 29,991 domains we have measured, 2%, disallow oai-searchbot at the site root in robots.txt. OAI-SearchBot builds the index that decides whether the operator's assistant cites a page.

Every domain here has removed itself from OpenAI's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

The measurement

608 of 29,991 measured domains (2%) disallow oai-searchbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block OAI-SearchBot", measured 2026-11-21. https://crawlcensus.com/blocked/oai-searchbot

The list 608 domains

RankDomainTLDAI access score
12,111 techpowerup.com .com 50
12,113 bostonherald.com .com 78
12,227 iol.pt .pt 80
12,377 delfi.ee .ee 57
12,441 novelfire.net .net 64
12,533 indystar.com .com 73
12,726 lalibre.be .be 83
12,743 extremetech.com .com 64
12,753 tvguide.com .com 70
12,841 stltoday.com .com 80
13,213 hellomagazine.com .com 86
13,277 mix.com .com 61
13,333 wtop.com .com 74
13,451 skyscrapercity.com .com 43
13,475 fotogramas.es .es 74
13,479 bnnbloomberg.ca .ca 81
13,569 blitz.gg .gg 64
13,579 barcelona.cat .cat 82
13,617 latribune.fr .fr 72
13,667 dispatch.com .com 74
13,792 sheknows.com .com 83
13,825 heraldscotland.com .com 81
13,860 francebleu.fr .fr 87
13,871 ilex.by .by 59
13,873 rappler.com .com 80
13,879 tennessean.com .com 74
13,987 mydealz.de .de 79
14,034 lrt.lt .lt 84
14,374 delfi.lv .lv 80
14,491 aerzteblatt.de .de 69
14,606 cnnturk.com .com 86
14,612 mopo.de .de 79
14,809 auto-motor-und-sport.de .de 81
14,826 rockpapershotgun.com .com 58
14,872 mz.de .de 70
14,961 townnews.com .com 75
15,174 pcgamesn.com .com 83
15,218 statesman.com .com 51
15,233 thieme-connect.com .com 39
15,260 birminghammail.co.uk .uk 47
15,269 gamesindustry.biz .biz 57
15,352 aaas.org .org 65
15,401 desmoinesregister.com .com 74
15,509 diariovasco.com .com 82
15,557 cincinnati.com .com 74
15,944 rtl.be .be 82
16,103 dweb.link .link 65
16,118 healio.com .com 69
16,209 ipfs.io .io 65
16,223 smashwords.com .com 58
16,254 unil.ch .ch 83
16,317 ilmattino.it .it 62
16,444 hometalk.com .com 63
16,520 hotukdeals.com .com 78
16,564 mapgenie.io .io 59
16,664 lightningmaps.org .org 51
16,738 cnbctv18.com .com 84
16,796 postandcourier.com .com 77
16,803 buffalonews.com .com 80
16,814 whattoexpect.com .com 65
16,903 adn.com .com 79
17,039 robbreport.com .com 64
17,042 northjersey.com .com 74
17,183 semafor.com .com 72
17,432 kyoto-np.co.jp .jp 82
17,455 twincities.com .com 81
17,469 palmbeachpost.com .com 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,619 nst.com.my .my 42
17,694 manager-magazin.de .de 63
17,886 agrarheute.com .com 75
18,147 volksstimme.de .de 68
18,195 film.ru .ru 61
18,280 hoerzu.de .de 45
18,309 mans.io .io 60
18,442 empireonline.com .com 62
18,595 trycloudflare.com .com 52
18,687 romatoday.it .it 67
18,782 pepper.pl .pl 76
18,863 dailynews.com .com 76
18,867 adressa.no .no 66
18,870 scotusblog.com .com 83
18,883 courier-journal.com .com 74
19,212 sex-studentki.studio .studio 57
19,283 denik.cz .cz 77
19,312 pinayflix.uk .uk 46
19,314 milanotoday.it .it 67
19,342 bold.dk .dk 58
19,398 ntvspor.net .net 84
19,411 clien.net .net 79
19,624 fukuishimbun.co.jp .jp 76
19,640 rssing.com .com 29
19,690 equidia.fr .fr 65
20,080 chroniclelive.co.uk .uk 47
20,251 elsevier.es .es 84
20,318 supersport.hr .hr 51
20,364 derwesten.de .de 83
20,575 artnews.com .com 80
20,871 lavenir.net .net 83
20,932 1kuji.com .com 69
20,958 pressdemocrat.com .com 80
21,139 ledevoir.com .com 70
21,267 seher.no .no 70
21,586 madison.com .com 80
21,707 khronos.org .org 68
21,736 slate.fr .fr 68
21,823 desy.de .de 62
22,007 vg247.com .com 58
22,028 distrowatch.com .com 44
22,216 airliners.net .net 44
22,365 petitesannonces.ch .ch 67
22,623 mcall.com .com 78
23,084 app.com .com 69
23,157 mlnba.fun .fun 54
24,014 askmen.com .com 47
24,028 canberratimes.com.au .au 79
24,102 knoxnews.com .com 74
24,190 jacksonville.com .com 74
24,199 sciam.com .com 78
24,413 irishmirror.ie .ie 47
24,453 dragcave.net .net 61
24,525 zin.ru .ru 41
24,567 newspicks.com .com 44
24,733 humanite.fr .fr 87
24,802 bristolpost.co.uk .uk 47
24,892 eldiariomontanes.es .es 82
24,961 miaminewtimes.com .com 73
25,031 franceculture.fr .fr 81
25,048 tokyo-np.co.jp .jp 81
25,115 pilotonline.com .com 79
25,184 kompasiana.com .com 85
25,391 americastestkitchen.com .com 80
25,451 omaha.com .com 80
25,478 franceinter.fr .fr 80
25,531 forebears.io .io 66
25,564 theadvocate.com .com 71
25,587 sfexaminer.com .com 69
25,670 cp24.com .com 81
25,839 delawareonline.com .com 74
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
26,005 people.inc .inc 64
26,114 democratandchronicle.com .com 73
26,284 daytondailynews.com .com 67
26,379 brisbanetimes.com.au .au 75
26,444 pap.pl .pl 39
26,474 mbta.com .com 64
26,567 spox.com .com 81
26,653 bas.bg .bg 82
26,853 tallahassee.com .com 74
26,935 fd.nl .nl 73
26,949 vocus.cc .cc 78
27,243 themirror.com .com 47
27,306 stylecaster.com .com 77
27,344 ga.de .de 80
27,361 westfalen-blatt.de .de 81
27,467 wfmz.com .com 81
27,696 staradvertiser.com .com 84
27,700 weser-kurier.de .de 87
27,723 geek.com .com 38
27,838 mtvuutiset.fi .fi 72
27,891 blogto.com .com 67
28,021 sr.ht .ht 55
28,616 amzn.com .com 40
28,626 openfoodfacts.org .org 59
28,642 nachrichten.at .at 82
28,678 thenational.scot .scot 81
28,767 mult-porno.team .team 56
28,794 eastbaytimes.com .com 79
28,930 championcross.jp .jp 62
28,950 acronymfinder.com .com 65
29,011 fraserinstitute.org .org 51
29,139 u-cursos.cl .cl 51
29,390 kompas.id .id 64
29,499 conjur.com.br .br 73
29,510 uvigo.es .es 56
29,526 sex-studentki.guru .guru 57
29,549 freiepresse.de .de 86
29,553 hankookilbo.com .com 84
29,634 westword.com .com 77
29,701 thieme-connect.de .de 79
29,821 providencejournal.com .com 73
29,884 techtudo.com.br .br 84
30,052 lasprovincias.es .es 82
30,074 yugipedia.com .com 58
30,196 youngchampion.jp .jp 62
30,237 clarionledger.com .com 74
30,263 androidheadlines.com .com 68
30,387 bsc.es .es 37
30,443 dobrzemieszkaj.pl .pl 61
30,457 fap-guru.name .name 56
30,703 wickedlocal.com .com 75
30,712 artforum.com .com 74
30,737 diariosur.es .es 82
30,773 rheinpfalz.de .de 77
30,810 kentonline.co.uk .uk 76
30,986 commercialappeal.com .com 74
31,010 mult-porno.world .world 56
31,020 flexmls.com .com 89
PreviousPage 2 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the OAI-SearchBot reference, the full census, and the JSON endpoint.