Blocklist · Allen Institute for AI

Sites that block AI2Bot

692 of the 29,991 domains we have measured, 2%, disallow ai2bot at the site root in robots.txt. AI2Bot collects content that may be used to train a model.

Every domain here has withheld its content from Allen Institute for AI's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

692 of 29,991 measured domains (2%) disallow ai2bot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block AI2Bot", measured 2026-11-22. https://crawlcensus.com/blocked/ai2bot

The list 692 domains

RankDomainTLDAI access score
43,593 mediamarkt.at .at 89
43,605 zonerama.com .com 30
43,974 ybw.com .com 87
43,986 nn.de .de 83
44,026 gainesville.com .com 66
44,077 airbnb.jp .jp 58
44,104 airbnb.nl .nl 58
44,183 barcablaugranes.com .com 68
44,252 braunschweiger-zeitung.de .de 65
44,255 cagesideseats.com .com 72
44,723 dailygazette.com .com 80
44,807 gofeminin.de .de 74
44,860 airbnb.co.za .za 58
44,874 news-leader.com .com 62
44,878 rtvnoord.nl .nl 73
44,908 mora.jp .jp 45
44,928 wgt.com .com 80
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,159 lansingstatejournal.com .com 62
45,272 rtvutrecht.nl .nl 73
45,316 eumetsat.int .int 80
45,349 wp.de .de 65
45,440 nrz.de .de 80
45,578 chugoku-np.co.jp .jp 80
45,604 beaconjournal.com .com 66
45,680 greenvilleonline.com .com 62
45,762 autozeitung.de .de 86
45,771 larioja.com .com 83
45,846 amazon.jp .jp 40
45,905 kob.com .com 84
45,989 aftenbladet.no .no 78
46,154 airbnb.com.co .co 58
46,169 airbnb.mx .mx 58
46,181 gardenersworld.com .com 82
46,231 registerguard.com .com 66
46,314 backpacker.com .com 78
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,635 wikihow.life .life 76
46,805 omroepwest.nl .nl 73
46,836 unibz.it .it 72
46,923 alisweb.org .org 39
46,949 schulferien.org .org 51
47,024 airbnb.co.kr .kr 59
47,029 lostarmour.info .info 51
47,173 eurogamer.de .de 57
47,285 bengo4.com .com 86
47,299 rtvdrenthe.nl .nl 73
47,492 kittl.com .com 91
47,526 allgaeuer-zeitung.de .de 81
47,659 nationalgeographic.de .de 82
47,796 topekalibraries.info .info 46
47,850 espinof.com .com 94
47,860 gpb.org .org 81
47,884 vcstar.com .com 62
47,909 moneyweek.com .com 86
47,930 diariodecadiz.es .es 67
48,046 trinidadexpress.com .com 74
48,074 grubstreet.com .com 81
48,102 airbnb.com.ar .ar 58
48,292 wellsky.com .com 74
48,480 xatakamovil.com .com 92
48,544 darkhorse.com .com 77
48,550 argusleader.com .com 62
48,589 airbnb.com.sg .sg 58
48,598 motorradonline.de .de 87
48,777 mtv.fi .fi 76
48,858 otz.de .de 65
49,061 posfie.com .com 73
49,065 ingramcontent.com .com 69
49,194 mailonsunday.co.uk .uk 61
49,245 startmycar.com .com 51
49,271 eltern.de .de 72
49,279 presstelegram.com .com 80
49,420 gds.it .it 77
49,447 businessinsider.es .es 87
49,501 sj-r.com .com 64
49,574 montgomeryadvertiser.com .com 62
49,695 pricesmart.com .com 61
49,802 pazar3.mk .mk 73
49,925 aamulehti.fi .fi 66
50,185 miricanvas.com .com 89
50,278 burlingtonfreepress.com .com 62
50,312 mujerhoy.com .com 80
50,338 ohtuleht.ee .ee 65
50,367 blipfoto.com .com 57
50,598 dailycal.org .org 74
50,732 vesty.co.il .il 91
50,933 nwaonline.com .com 73
50,960 hot.com .com 67
50,982 nrtool.st .st 54
PreviousPage 4 of 4

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the AI2Bot reference, the full census, and the JSON endpoint.