Blocklist · Allen Institute for AI

Sites that block AI2Bot

692 of the 29,991 domains we have measured, 2%, disallow ai2bot at the site root in robots.txt. AI2Bot collects content that may be used to train a model.

Every domain here has withheld its content from Allen Institute for AI's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

692 of 29,991 measured domains (2%) disallow ai2bot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block AI2Bot", measured 2026-11-22. https://crawlcensus.com/blocked/ai2bot

The list 692 domains

RankDomainTLDAI access score
9,339 detroitnews.com .com 68
9,388 law.com .com 68
9,885 wral.com .com 87
10,289 interestingengineering.com .com 87
10,335 windowscentral.com .com 85
10,345 inews.co.uk .uk 67
10,400 morgenpost.de .de 81
10,432 expansion.com .com 81
10,470 babycenter.com .com 62
10,501 ocregister.com .com 80
10,585 anyflip.com .com 56
10,727 radiotimes.com .com 80
10,885 livestrong.com .com 80
10,894 augsburger-allgemeine.de .de 81
10,934 huffingtonpost.fr .fr 81
11,005 airbnb.co.uk .uk 58
11,107 bibliocommons.com .com 85
11,130 lexpress.fr .fr 86
11,147 intechopen.com .com 67
11,200 aftenposten.no .no 76
11,216 nola.com .com 64
11,217 businessinsider.de .de 76
11,349 posteo.de .de 67
11,390 pantone.com .com 78
11,405 svd.se .se 78
11,457 wnp.pl .pl 65
11,516 cults3d.com .com 91
11,659 abendblatt.de .de 65
12,113 bostonherald.com .com 78
12,294 ally.com .com 94
12,297 slideserve.com .com 82
12,377 delfi.ee .ee 57
12,533 indystar.com .com 73
12,550 kiplinger.com .com 85
12,607 joinhandshake.com .com 87
12,726 lalibre.be .be 83
12,743 extremetech.com .com 64
12,797 golem.de .de 65
12,799 curbed.com .com 73
12,841 stltoday.com .com 80
12,924 syfy.com .com 80
12,972 noticiasaominuto.com .com 85
12,990 nybooks.com .com 80
13,091 elle.fr .fr 69
13,131 ranker.com .com 78
13,169 csoonline.com .com 86
13,237 clicrbs.com.br .br 80
13,355 marieclaire.com .com 85
13,380 flourish.studio .studio 92
13,579 barcelona.cat .cat 82
13,617 latribune.fr .fr 72
13,660 thrillist.com .com 70
13,667 dispatch.com .com 74
13,783 creativebloq.com .com 87
13,825 heraldscotland.com .com 81
13,858 flashscore.co.za .za 70
13,860 francebleu.fr .fr 87
13,871 ilex.by .by 59
13,879 tennessean.com .com 74
13,882 vermont.gov .gov 73
13,937 songkick.com .com 66
14,053 androidcentral.com .com 83
14,079 wccftech.com .com 80
14,090 simon.com .com 31
14,168 ncaa.com .com 85
14,374 delfi.lv .lv 80
14,404 cumhuriyet.com.tr .tr 84
14,612 mopo.de .de 79
14,709 dazeddigital.com .com 78
14,733 waz.de .de 82
14,809 auto-motor-und-sport.de .de 81
14,826 rockpapershotgun.com .com 58
14,961 townnews.com .com 75
15,174 pcgamesn.com .com 83
15,218 statesman.com .com 51
15,269 gamesindustry.biz .biz 57
15,311 mcmaster.com .com 69
15,321 coursehero.com .com 76
15,372 letudiant.fr .fr 81
15,401 desmoinesregister.com .com 74
15,476 computerbild.de .de 81
15,509 diariovasco.com .com 82
15,557 cincinnati.com .com 74
15,686 postimees.ee .ee 66
15,944 rtl.be .be 82
16,082 t3n.de .de 78
16,086 nintendo.co.jp .jp 50
16,118 healio.com .com 69
16,134 homesandgardens.com .com 84
16,252 vivino.com .com 82
16,254 unil.ch .ch 83
16,290 duden.de .de 71
16,428 e24.no .no 78
16,564 mapgenie.io .io 59
16,664 lightningmaps.org .org 51
16,698 airbnb.com.br .br 58
16,783 taipeitimes.com .com 66
16,795 vozpopuli.com .com 87
16,796 postandcourier.com .com 77
16,803 buffalonews.com .com 80
16,814 whattoexpect.com .com 65
17,042 northjersey.com .com 74
17,183 semafor.com .com 72
17,455 twincities.com .com 81
17,469 palmbeachpost.com .com 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,694 manager-magazin.de .de 63
17,886 agrarheute.com .com 75
17,890 directoalpaladar.com .com 92
17,936 cinemablend.com .com 87
18,071 flashscore.co.uk .uk 71
18,280 hoerzu.de .de 45
18,308 preyproject.com .com 93
18,442 empireonline.com .com 62
18,489 elcolombiano.com .com 83
18,545 prozeny.cz .cz 72
18,608 sensacine.com .com 81
18,863 dailynews.com .com 76
18,867 adressa.no .no 66
18,870 scotusblog.com .com 83
18,883 courier-journal.com .com 74
18,894 courrierinternational.com .com 83
19,189 airbnb.co.in .in 58
19,250 beatstars.com .com 51
19,283 denik.cz .cz 77
19,287 superbet.pl .pl 50
19,631 spectrumlocalnews.com .com 45
19,690 equidia.fr .fr 65
19,860 zawya.com .com 80
19,891 musicradar.com .com 87
20,081 flashscore.com.tr .tr 70
20,318 supersport.hr .hr 51
20,362 thedodo.com .com 70
20,364 derwesten.de .de 83
20,419 01net.com .com 93
20,460 mt.co.kr .kr 93
20,481 flashscore.pt .pt 72
20,556 bravotv.com .com 74
20,676 haufe.de .de 76
20,718 itpro.com .com 83
20,761 nlc.hu .hu 85
20,847 gardeningknowhow.com .com 83
20,858 flashscore.vn .vn 72
20,871 lavenir.net .net 83
20,958 pressdemocrat.com .com 80
21,004 femina.hu .hu 82
21,038 mainpost.de .de 80
21,231 epochtimes.com .com 85
21,267 seher.no .no 70
21,481 futureplc.com .com 86
21,566 whowhatwear.com .com 82
21,586 madison.com .com 80
21,809 cyclingnews.com .com 85
21,823 desy.de .de 62
21,828 wallpaper.com .com 85
22,007 vg247.com .com 58
22,016 canva.in .in 86
22,149 airbnb.ca .ca 58
22,596 flashscore.gr .gr 72
22,623 mcall.com .com 78
23,084 app.com .com 69
23,613 nordbayern.de .de 81
23,875 imore.com .com 81
23,964 pcwelt.de .de 78
24,000 icecat.biz .biz 44
24,014 askmen.com .com 47
24,102 knoxnews.com .com 74
24,157 whathifi.com .com 87
24,190 jacksonville.com .com 74
24,468 flashscore.de .de 71
24,525 zin.ru .ru 41
24,714 occrp.org .org 62
24,842 sdstate.edu .edu 74
24,882 uol.com .com 82
24,892 eldiariomontanes.es .es 82
24,934 svtplay.se .se 78
24,961 miaminewtimes.com .com 73
25,007 digitalcameraworld.com .com 86
25,031 franceculture.fr .fr 81
25,048 tokyo-np.co.jp .jp 81
25,065 ny1.com .com 68
25,115 pilotonline.com .com 79
25,237 flashscorekz.com .com 70
25,451 omaha.com .com 80
25,478 franceinter.fr .fr 80
25,564 theadvocate.com .com 71
25,587 sfexaminer.com .com 69
25,653 livesport.com .com 69
25,677 flashscore.in .in 70
25,776 zinio.com .com 76
25,796 shopltk.com .com 83
25,800 trendencias.com .com 94
25,839 delawareonline.com .com 74
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
26,114 democratandchronicle.com .com 73
26,210 kstp.com .com 83
26,240 yogajournal.com .com 77
26,244 superbet.rs .rs 50
PreviousPage 2 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the AI2Bot reference, the full census, and the JSON endpoint.