Blocklist · Allen Institute for AI

Sites that block AI2Bot

692 of the 29,991 domains we have measured, 2%, disallow ai2bot at the site root in robots.txt. AI2Bot collects content that may be used to train a model.

Every domain here has withheld its content from Allen Institute for AI's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

692 of 29,991 measured domains (2%) disallow ai2bot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block AI2Bot", measured 2026-11-22. https://crawlcensus.com/blocked/ai2bot

The list 692 domains

RankDomainTLDAI access score
26 amazon.com .com 40
54 tiktok.com .com 49
62 msn.com .com 41
121 amazonvideo.com .com 55
190 cnn.com .com 50
260 launchpad.net .net 52
318 amazon.co.uk .uk 40
335 ea.com .com 76
377 amazon.de .de 40
494 amazon.co.jp .jp 40
571 uol.com.br .br 82
609 amazon.fr .fr 40
617 unesco.org .org 90
623 amazon.co.za .za 40
627 dailymail.co.uk .uk 61
628 nintendo.com .com 61
635 amazon.ca .ca 40
659 amazon.in .in 40
662 cnet.com .com 79
671 lemonde.fr .fr 81
681 dailymail.com .com 61
682 primevideo.com .com 71
706 amazon.it .it 40
725 amazon.es .es 40
749 airbnb.com .com 36
763 bild.de .de 81
812 amazon.com.au .au 40
815 theverge.com .com 77
833 amazon.com.br .br 40
955 corriere.it .it 63
983 elmundo.es .es 84
1,019 theconversation.com .com 62
1,032 amazon.com.mx .mx 40
1,042 spiegel.de .de 64
1,056 marca.com .com 85
1,074 flashscore.com .com 73
1,098 franceinfo.fr .fr 60
1,105 cookpad.com .com 73
1,177 pimpbunny.com .com 56
1,189 aftonbladet.se .se 77
1,208 vg.no .no 79
1,223 furaffinity.net .net 39
1,232 pandora.com .com 53
1,300 f95zone.to .to 59
1,305 dw.com .com 42
1,374 novinky.cz .cz 76
1,376 dagbladet.no .no 74
1,394 usnews.com .com 83
1,409 iltalehti.fi .fi 58
1,624 flashscore.mobi .mobi 69
1,640 flashscore.fr .fr 71
1,676 livescore.cz .cz 69
1,831 zdnet.com .com 76
1,835 wikihow.com .com 76
1,984 themoviedb.org .org 74
1,993 ign.com .com 62
2,006 amazon.eg .eg 40
2,010 mashable.com .com 61
2,067 proofpoint.com .com 88
2,215 techradar.com .com 87
2,325 nrk.no .no 69
2,346 slate.com .com 80
2,351 diretta.it .it 70
2,423 pcmag.com .com 63
2,585 livescience.com .com 85
2,691 newscientist.com .com 76
2,723 yumpu.com .com 30
2,786 chicagotribune.com .com 79
2,838 mastodon.social .social 49
2,917 faz.net .net 39
2,975 imageshack.us .us 71
3,179 crunchbase.com .com 71
3,198 metro.co.uk .uk 59
3,276 myanimelist.net .net 70
3,286 flipboard.com .com 64
3,331 tomshardware.com .com 87
3,342 lifehacker.com .com 65
3,368 nzherald.co.nz .nz 56
3,371 nymag.com .com 74
3,437 tagesspiegel.de .de 74
3,499 amazon.se .se 40
3,520 amazon.ae .ae 40
3,542 orf.at .at 69
3,576 abc.es .es 81
3,595 politico.eu .eu 62
3,598 amazon.nl .nl 40
3,771 yle.fi .fi 85
3,800 ebsco.com .com 65
3,869 pcworld.com .com 84
3,873 space.com .com 85
3,963 nos.nl .nl 69
4,065 seekingalpha.com .com 91
4,161 kompas.com .com 77
4,181 sbs.com.au .au 83
4,190 zoominfo.com .com 90
4,354 computerworld.com .com 83
4,407 20minutos.es .es 87
4,512 amazon.com.tr .tr 40
4,551 yna.co.kr .kr 87
4,929 onlamp.com .com 80
5,026 pcgamer.com .com 85
5,037 imageshack.com .com 71
5,146 everydayhealth.com .com 60
5,158 mercurynews.com .com 78
5,268 ora.com .com 94
5,272 depositphotos.com .com 63
5,297 chinatimes.com .com 64
5,310 kotaku.com .com 89
5,357 larazon.es .es 81
5,363 allocine.fr .fr 81
5,401 mediamarkt.de .de 90
5,490 superbet.ro .ro 54
5,547 amazon.ie .ie 40
5,678 dn.se .se 60
5,684 theepochtimes.com .com 80
5,714 francetvinfo.fr .fr 53
5,726 chefkoch.de .de 81
5,752 sankei.com .com 76
5,775 tv2.no .no 70
5,846 runescape.wiki .wiki 81
5,874 hs.fi .fi 68
5,879 cio.com .com 86
6,064 delfi.lt .lt 83
6,089 azcentral.com .com 74
6,122 jeuxvideo.com .com 71
6,125 denverpost.com .com 78
6,159 instiz.net .net 58
6,180 tf1info.fr .fr 84
6,185 trustedshops.com .com 87
6,228 sbnation.com .com 69
6,246 eurogamer.net .net 57
6,288 vulture.com .com 72
6,352 tv2.dk .dk 74
6,353 bbcgoodfood.com .com 84
6,356 freep.com .com 73
6,361 ici.fr .fr 87
6,377 gamesradar.com .com 85
6,512 telegram.com .com 73
6,513 sandiegouniontribune.com .com 78
6,522 eater.com .com 59
6,610 is.fi .fi 68
6,702 infoworld.com .com 83
6,726 rtbf.be .be 67
6,793 svt.se .se 66
6,808 chunichi.co.jp .jp 71
6,816 maxroll.gg .gg 59
6,822 elcorreo.com .com 80
6,895 togetter.com .com 73
6,910 xataka.com .com 94
6,922 livescore.in .in 71
6,948 nicematin.com .com 77
7,095 seznamzpravy.cz .cz 71
7,106 nrc.nl .nl 84
7,127 hani.co.kr .kr 67
7,178 sanspo.com .com 77
7,201 game8.jp .jp 72
7,239 gala.de .de 76
7,247 nishinippon.co.jp .jp 69
7,275 baltimoresun.com .com 76
7,372 15min.lt .lt 74
7,396 outsideonline.com .com 79
7,455 amazon.sg .sg 40
7,465 nationalgeographic.org .org 82
7,562 nettavisen.no .no 63
7,613 filmstarts.de .de 81
7,647 sporza.be .be 87
7,780 purepeople.com .com 85
7,836 sport.cz .cz 71
7,856 popsugar.com .com 75
7,863 refinery29.com .com 78
7,925 adorocinema.com .com 81
7,981 afr.com .com 72
8,013 super.cz .cz 75
8,094 nefisyemektarifleri.com .com 76
8,110 tudogostoso.com.br .br 83
8,207 flashscore.pl .pl 71
8,268 litmarket.ru .ru 87
8,364 eredmenyek.com .com 71
8,380 veikkaus.fi .fi 67
8,398 livesport.cz .cz 71
8,435 thecut.com .com 84
8,450 complex.com .com 91
8,484 sciencenews.org .org 86
8,599 soccerway.com .com 73
8,710 rezultati.com .com 71
8,830 portalsamorzadowy.pl .pl 69
8,866 theweek.com .com 84
8,897 rynekzdrowia.pl .pl 64
8,899 flashscore.co.id .id 70
8,922 kupi.cz .cz 84
9,007 flashscore.co .co 69
9,068 airbnb.fr .fr 58
9,069 wbur.org .org 81
9,072 flashscore.ua .ua 70
9,094 flashscore.ro .ro 70
9,149 forth.gr .gr 60
9,196 flashscore.es .es 69
9,302 macworld.com .com 85
9,317 flashscore.com.ve .ve 69
9,320 flashscore.pe .pe 70
Page 1 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the AI2Bot reference, the full census, and the JSON endpoint.