Blocklist · Meta

Sites that block Meta-WebIndexer

402 of the 29,991 domains we have measured, 1%, disallow meta-webindexer at the site root in robots.txt. Meta-WebIndexer builds the index that decides whether the operator's assistant cites a page.

Every domain here has removed itself from Meta's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

The measurement

402 of 29,991 measured domains (1%) disallow meta-webindexer at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Meta-WebIndexer", measured 2026-11-21. https://crawlcensus.com/blocked/meta-webindexer

The list 402 domains

RankDomainTLDAI access score
24,892 eldiariomontanes.es .es 82
24,946 vogue.fr .fr 81
24,961 miaminewtimes.com .com 73
24,977 wired.it .it 83
24,983 sn.at .at 70
25,007 digitalcameraworld.com .com 86
25,031 franceculture.fr .fr 81
25,115 pilotonline.com .com 79
25,133 myrecipes.com .com 77
25,478 franceinter.fr .fr 80
25,839 delawareonline.com .com 74
25,855 deutsche-rentenversicherung.de .de 79
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
26,114 democratandchronicle.com .com 73
26,249 guitarworld.com .com 84
26,637 brigitte.de .de 79
26,853 tallahassee.com .com 74
27,315 fourfourtwo.com .com 84
27,723 geek.com .com 38
27,881 cas.org .org 89
28,021 sr.ht .ht 55
28,320 laptopmag.com .com 87
28,616 amzn.com .com 40
28,626 openfoodfacts.org .org 59
28,775 vogue.de .de 65
28,794 eastbaytimes.com .com 79
28,849 unitel.bo .bo 75
29,048 gq-magazin.de .de 66
29,139 u-cursos.cl .cl 51
29,390 kompas.id .id 64
29,634 westword.com .com 77
29,639 edmontonjournal.com .com 71
29,701 thieme-connect.de .de 79
29,821 providencejournal.com .com 73
30,004 sfstandard.com .com 72
30,052 lasprovincias.es .es 82
30,237 clarionledger.com .com 74
30,703 wickedlocal.com .com 75
30,737 diariosur.es .es 82
30,810 kentonline.co.uk .uk 76
30,986 commercialappeal.com .com 74
31,020 flexmls.com .com 89
31,023 phoenixnewtimes.com .com 71
31,032 howlongtobeat.com .com 49
31,138 ideal.es .es 82
31,149 idealhome.co.uk .uk 82
31,229 bookmeter.com .com 68
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,638 marieclaire.co.uk .uk 84
31,786 thespec.com .com 64
31,850 avsforum.com .com 43
31,857 weawow.com .com 65
31,914 springerprofessional.de .de 60
31,934 comicbookmovie.com .com 78
32,141 novakidschool.com .com 91
32,279 custommapposter.com .com 77
32,388 rgj.com .com 73
32,437 hoopshype.com .com 70
32,498 toptenreviews.com .com 87
32,561 skapiec.pl .pl 77
32,579 dallasobserver.com .com 77
32,617 flo.com.tr .tr 52
32,790 tatler.com .com 82
32,901 stocktitan.net .net 94
32,918 laverdad.es .es 82
34,012 womanandhome.com .com 85
34,023 followfollow.com .com 73
34,092 portafolio.co .co 89
34,097 vogue.es .es 65
34,309 unionleader.com .com 63
34,750 mailplus.co.uk .uk 44
34,847 sydsvenskan.se .se 52
34,973 naplesnews.com .com 73
34,986 ruhrnachrichten.de .de 75
35,062 dailypress.com .com 79
35,543 news-journalonline.com .com 73
35,759 houseandgarden.co.uk .uk 81
35,760 montgomerycountymd.gov .gov 65
36,120 elcomercio.es .es 82
36,237 saltwire.com .com 70
36,334 ytmnd.com .com 41
36,436 liquor.com .com 77
36,564 whatfontis.com .com 91
36,646 cyclingweekly.com .com 85
37,062 koreaboo.com .com 91
37,821 golfmonthly.com .com 83
38,007 wired.jp .jp 78
38,148 news-press.com .com 73
38,327 inforum.com .com 70
38,409 igt.com .com 48
38,504 bundesarchiv.de .de 77
38,739 thueringer-allgemeine.de .de 82
38,876 elnortedecastilla.es .es 82
38,945 postype.com .com 81
39,206 futurezone.de .de 86
39,349 hoy.es .es 82
39,471 vwvortex.com .com 43
39,519 vinepair.com .com 84
39,706 indiehackers.com .com 54
39,806 mercateo.com .com 48
39,824 werstreamt.es .es 59
40,339 kompas.tv .tv 76
40,392 vanguardia.com.mx .mx 84
40,638 main-echo.de .de 64
40,796 elpasotimes.com .com 73
40,874 grid.id .id 64
40,876 cntraveller.in .in 81
41,098 gqmagazine.fr .fr 82
41,140 rixos.com .com 69
41,290 theweek.co.uk .uk 84
41,647 cjonline.com .com 73
41,760 dailybreeze.com .com 76
41,872 vanityfair.fr .fr 66
41,947 vogue.co.jp .jp 65
41,958 theledger.com .com 74
42,076 iex.nl .nl 48
42,160 tcpalm.com .com 73
42,260 rcgroups.com .com 41
42,262 advrider.com .com 41
42,582 decanter.com .com 85
42,614 citizen-times.com .com 74
42,925 pixta.jp .jp 78
42,981 viewbug.com .com 83
43,055 deepdreamgenerator.com .com 79
43,268 candlepowerforums.com .com 61
43,302 seacoastonline.com .com 69
43,346 aerztezeitung.de .de 90
43,587 admagazine.fr .fr 81
43,659 medonet.pl .pl 84
43,974 ybw.com .com 87
43,983 teambelgium.net .net 59
44,026 gainesville.com .com 66
44,252 braunschweiger-zeitung.de .de 65
44,299 theprovince.com .com 71
44,807 gofeminin.de .de 74
44,874 news-leader.com .com 62
44,878 rtvnoord.nl .nl 73
44,928 wgt.com .com 80
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,159 lansingstatejournal.com .com 62
45,240 vogue.mx .mx 65
45,272 rtvutrecht.nl .nl 73
45,349 wp.de .de 65
45,440 nrz.de .de 80
45,578 chugoku-np.co.jp .jp 80
45,604 beaconjournal.com .com 66
45,644 jewishjournal.com .com 83
45,680 greenvilleonline.com .com 62
45,771 larioja.com .com 83
45,846 amazon.jp .jp 40
45,918 revistagq.com .com 81
46,168 bizapedia.com .com 60
46,231 registerguard.com .com 66
46,250 openarchieven.nl .nl 87
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,594 okazii.ro .ro 67
46,635 wikihow.life .life 76
46,805 omroepwest.nl .nl 73
46,923 alisweb.org .org 39
47,173 eurogamer.de .de 57
47,299 rtvdrenthe.nl .nl 73
47,624 lacucinaitaliana.it .it 81
47,741 salzburg24.at .at 85
47,796 topekalibraries.info .info 46
47,884 vcstar.com .com 62
47,909 moneyweek.com .com 86
48,010 lfpress.com .com 87
48,129 komputerswiat.pl .pl 82
48,444 vogue.com.tw .tw 81
48,513 airnav.com .com 53
48,550 argusleader.com .com 62
48,858 otz.de .de 65
48,859 castro.fm .fm 67
49,059 citypopulation.de .de 54
49,061 posfie.com .com 73
49,067 gqitalia.it .it 64
49,194 mailonsunday.co.uk .uk 61
49,271 eltern.de .de 72
49,279 presstelegram.com .com 80
49,343 hd.se .se 51
49,489 sinemalar.com .com 55
49,501 sj-r.com .com 64
49,518 tiger-algebra.com .com 80
49,574 montgomeryadvertiser.com .com 62
49,925 aamulehti.fi .fi 66
50,003 europeanfilmgateway.eu .eu 65
50,085 selectra.info .info 90
50,278 burlingtonfreepress.com .com 62
50,312 mujerhoy.com .com 80
50,426 bolshoyvopros.ru .ru 70
50,442 ottawa.ca .ca 46
50,492 financialfilings.com .com 94
50,684 gq.com.mx .mx 72
50,727 namur.be .be 67
PreviousPage 2 of 3Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Meta-WebIndexer reference, the full census, and the JSON endpoint.