Blocklist · Allen Institute for AI

Sites that block AI2Bot

692 of the 29,991 domains we have measured, 2%, disallow ai2bot at the site root in robots.txt. AI2Bot collects content that may be used to train a model.

Every domain here has withheld its content from Allen Institute for AI's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

692 of 29,991 measured domains (2%) disallow ai2bot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block AI2Bot", measured 2026-11-22. https://crawlcensus.com/blocked/ai2bot

The list 692 domains

RankDomainTLDAI access score
26,249 guitarworld.com .com 84
26,284 daytondailynews.com .com 67
26,474 mbta.com .com 64
26,637 brigitte.de .de 79
26,806 golf.com .com 88
26,853 tallahassee.com .com 74
26,935 fd.nl .nl 73
26,963 premiumoutlets.com .com 31
26,970 airbnb.de .de 57
27,078 mdzol.com .com 61
27,136 purepeople.com.br .br 85
27,182 ultimahora.es .es 83
27,248 oreillynet.com .com 94
27,256 amny.com .com 83
27,315 fourfourtwo.com .com 84
27,361 westfalen-blatt.de .de 81
27,467 wfmz.com .com 81
27,723 geek.com .com 38
27,756 beatstars.net .net 51
27,838 mtvuutiset.fi .fi 72
27,946 trustedshops.de .de 73
27,999 earth.com .com 80
28,170 suedkurier.de .de 81
28,220 icecat.se .se 52
28,290 icecat.dk .dk 56
28,320 laptopmag.com .com 87
28,616 amzn.com .com 40
28,678 thenational.scot .scot 81
28,792 uesp.net .net 55
28,794 eastbaytimes.com .com 79
28,930 championcross.jp .jp 62
29,115 gp.se .se 78
29,139 u-cursos.cl .cl 51
29,323 goop.com .com 66
29,390 kompas.id .id 64
29,402 mmafighting.com .com 68
29,499 conjur.com.br .br 73
29,553 hankookilbo.com .com 84
29,572 flashscore.com.mx .mx 70
29,634 westword.com .com 77
29,692 piliapp.com .com 65
29,821 providencejournal.com .com 73
30,004 sfstandard.com .com 72
30,052 lasprovincias.es .es 82
30,074 yugipedia.com .com 58
30,196 youngchampion.jp .jp 62
30,237 clarionledger.com .com 74
30,399 spoton.com .com 94
30,443 dobrzemieszkaj.pl .pl 61
30,619 spectrumnews1.com .com 45
30,703 wickedlocal.com .com 75
30,737 diariosur.es .es 82
30,810 kentonline.co.uk .uk 76
30,974 techadvisor.com .com 85
30,986 commercialappeal.com .com 74
31,003 maxim.com .com 87
31,023 phoenixnewtimes.com .com 71
31,032 howlongtobeat.com .com 49
31,138 ideal.es .es 82
31,149 idealhome.co.uk .uk 82
31,176 edp24.co.uk .uk 79
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,352 sportsbusinessjournal.com .com 53
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,638 marieclaire.co.uk .uk 84
31,782 mastodon.online .online 49
31,786 thespec.com .com 64
31,871 diariodesevilla.es .es 84
32,032 prx.org .org 74
32,169 journaldugeek.com .com 91
32,267 theargus.co.uk .uk 79
32,388 rgj.com .com 73
32,437 hoopshype.com .com 70
32,460 saturn.de .de 91
32,498 toptenreviews.com .com 87
32,579 dallasobserver.com .com 77
32,631 baynews9.com .com 68
32,706 marianne.net .net 67
32,713 opovo.com.br .br 85
32,718 athletic.net .net 68
32,803 oxfordmail.co.uk .uk 79
32,806 tvnet.lv .lv 79
32,918 laverdad.es .es 82
33,254 aan.com .com 74
33,312 airbnb.es .es 59
33,349 qq8893.com .com 68
33,482 airbnb.it .it 58
33,494 theanalyst.com .com 88
33,562 myporntube.com .com 86
33,568 icecat.it .it 44
33,644 tv4.se .se 76
33,738 ruv.is .is 59
33,793 observer.co.uk .uk 74
33,931 lancasteronline.com .com 79
33,934 carlsen.de .de 71
33,971 airbnb.com.au .au 58
34,012 womanandhome.com .com 85
34,309 unionleader.com .com 63
34,387 breakingnews.ie .ie 71
34,445 monoskop.org .org 43
34,491 sfweekly.com .com 74
34,750 mailplus.co.uk .uk 44
34,822 milanofinanza.it .it 82
34,973 naplesnews.com .com 73
34,986 ruhrnachrichten.de .de 75
35,058 gaoding.com .com 62
35,062 dailypress.com .com 79
35,198 historyextra.com .com 82
35,364 road.cc .cc 84
35,404 mapadeapellidos.com .com 75
35,543 news-journalonline.com .com 73
35,763 penguin.de .de 88
35,800 wn.de .de 80
35,933 hrs.com .com 82
36,120 elcomercio.es .es 82
36,334 ytmnd.com .com 41
36,356 elgaronline.com .com 45
36,539 nd-aktuell.de .de 78
36,575 dailyecho.co.uk .uk 81
36,630 roadsideamerica.com .com 72
36,646 cyclingweekly.com .com 85
36,803 tennis.com .com 59
37,154 r777.net .net 51
37,297 storyblocks.com .com 70
37,547 bt.no .no 77
37,598 justapinch.com .com 71
37,821 golfmonthly.com .com 83
38,129 11freunde.de .de 58
38,148 news-press.com .com 73
38,247 moviemaker.com .com 90
38,327 inforum.com .com 70
38,406 icecat.us .us 61
38,409 igt.com .com 48
38,476 jarir.com .com 77
38,507 bournemouthecho.co.uk .uk 79
38,519 kochbar.de .de 62
38,668 stimme.de .de 72
38,739 thueringer-allgemeine.de .de 82
38,851 financemagnates.com .com 90
38,876 elnortedecastilla.es .es 82
38,945 postype.com .com 81
39,206 futurezone.de .de 86
39,210 thecentersquare.com .com 78
39,349 hoy.es .es 82
39,350 thehighline.org .org 77
39,369 santafenewmexican.com .com 76
39,393 zakzak.co.jp .jp 72
39,657 korben.info .info 77
39,824 werstreamt.es .es 59
40,184 estantevirtual.com.br .br 91
40,219 thegrio.com .com 89
40,317 iza.ne.jp .jp 79
40,328 bntu.by .by 80
40,339 kompas.tv .tv 76
40,512 thetelegraphandargus.co.uk .uk 78
40,638 main-echo.de .de 64
40,713 dwds.de .de 64
40,726 flashscoreusa.com .com 71
40,728 mediamarkt.pl .pl 89
40,741 theblast.com .com 92
40,796 elpasotimes.com .com 73
40,830 xink.io .io 78
40,868 sayweee.com .com 72
40,874 grid.id .id 64
41,121 lancashiretelegraph.co.uk .uk 78
41,290 theweek.co.uk .uk 84
41,295 newson6.com .com 82
41,336 archinect.com .com 68
41,394 capitalizemytitle.com .com 71
41,519 greenme.it .it 82
41,542 oefb.at .at 50
41,603 lci.fr .fr 84
41,614 thebulwark.com .com 54
41,647 cjonline.com .com 73
41,713 roanoke.com .com 80
41,721 canarias7.es .es 83
41,736 eadt.co.uk .uk 79
41,760 dailybreeze.com .com 76
41,826 defcon.org .org 66
41,958 theledger.com .com 74
42,054 lecker.de .de 88
42,076 iex.nl .nl 48
42,112 news9.com .com 85
42,113 glasgowtimes.co.uk .uk 79
42,160 tcpalm.com .com 73
42,209 stream.cz .cz 81
42,582 decanter.com .com 85
42,614 citizen-times.com .com 74
42,651 computerwoche.de .de 85
42,682 opinautos.com .com 52
42,698 sol.no .no 74
42,862 horseracingnation.com .com 72
42,960 ceskenoviny.cz .cz 68
43,132 jtbc.co.kr .kr 70
43,207 tv4play.se .se 77
43,302 seacoastonline.com .com 69
43,310 3djuegos.com .com 93
43,548 tvmovie.de .de 89
PreviousPage 3 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the AI2Bot reference, the full census, and the JSON endpoint.