Blocklist · Meta

Sites that block Meta-ExternalFetcher

553 of the 29,991 domains we have measured, 2%, disallow meta-externalfetcher at the site root in robots.txt. Meta-ExternalFetcher fetches a page live when a user asks the assistant for it.

Every domain here has removed itself from Meta's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

Meta documents that this agent may ignore robots.txt, so a Disallow here records the site's stated preference rather than an enforced block.
The measurement

553 of 29,991 measured domains (2%) disallow meta-externalfetcher at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Meta-ExternalFetcher", measured 2026-11-21. https://crawlcensus.com/blocked/meta-externalfetcher

The list 553 domains

RankDomainTLDAI access score
12,441 novelfire.net .net 64
12,533 indystar.com .com 73
12,542 epicurious.com .com 64
12,726 lalibre.be .be 83
12,743 extremetech.com .com 64
12,799 curbed.com .com 73
12,972 noticiasaominuto.com .com 85
13,398 gq-magazine.co.uk .uk 81
13,419 vancouversun.com .com 71
13,569 blitz.gg .gg 64
13,617 latribune.fr .fr 72
13,660 thrillist.com .com 70
13,667 dispatch.com .com 74
13,668 torontosun.com .com 71
13,792 sheknows.com .com 83
13,825 heraldscotland.com .com 81
13,860 francebleu.fr .fr 87
13,871 ilex.by .by 59
13,873 rappler.com .com 80
13,879 tennessean.com .com 74
14,168 ncaa.com .com 85
14,374 delfi.lv .lv 80
14,612 mopo.de .de 79
14,809 auto-motor-und-sport.de .de 81
14,826 rockpapershotgun.com .com 58
14,862 canada.com .com 71
14,873 rareanimes.mov .mov 72
15,080 adl.org .org 87
15,174 pcgamesn.com .com 83
15,218 statesman.com .com 51
15,233 thieme-connect.com .com 39
15,269 gamesindustry.biz .biz 57
15,372 letudiant.fr .fr 81
15,401 desmoinesregister.com .com 74
15,509 diariovasco.com .com 82
15,557 cincinnati.com .com 74
15,944 rtl.be .be 82
16,082 t3n.de .de 78
16,103 dweb.link .link 65
16,209 ipfs.io .io 65
16,223 smashwords.com .com 58
16,254 unil.ch .ch 83
16,256 runnerspace.com .com 47
16,377 decider.com .com 79
16,564 mapgenie.io .io 59
16,664 lightningmaps.org .org 51
16,814 whattoexpect.com .com 65
17,039 robbreport.com .com 64
17,042 northjersey.com .com 74
17,183 semafor.com .com 72
17,266 letemps.ch .ch 74
17,420 jsbin.com .com 53
17,432 kyoto-np.co.jp .jp 82
17,455 twincities.com .com 81
17,469 palmbeachpost.com .com 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,619 nst.com.my .my 42
17,833 novelphoenix.com .com 61
17,886 agrarheute.com .com 75
18,309 mans.io .io 60
18,442 empireonline.com .com 62
18,522 thenews.com.pk .pk 65
18,595 trycloudflare.com .com 52
18,687 romatoday.it .it 67
18,799 ironman.com .com 75
18,863 dailynews.com .com 76
18,883 courier-journal.com .com 74
18,894 courrierinternational.com .com 83
19,312 pinayflix.uk .uk 46
19,314 milanotoday.it .it 67
19,342 bold.dk .dk 58
19,624 fukuishimbun.co.jp .jp 76
19,762 desixxx.blog .blog 65
19,860 zawya.com .com 80
19,924 jansatta.com .com 86
19,941 ottawacitizen.com .com 71
20,318 supersport.hr .hr 51
20,362 thedodo.com .com 70
20,575 artnews.com .com 80
20,871 lavenir.net .net 83
20,958 pressdemocrat.com .com 80
21,038 mainpost.de .de 80
21,139 ledevoir.com .com 70
21,267 seher.no .no 70
21,685 calgaryherald.com .com 71
21,823 desy.de .de 62
22,007 vg247.com .com 58
22,365 petitesannonces.ch .ch 67
22,623 mcall.com .com 78
22,938 congobet.net .net 45
23,084 app.com .com 69
23,157 mlnba.fun .fun 54
23,237 bet261.mg .mg 45
23,298 fuegodevida.com .com 58
24,014 askmen.com .com 47
24,102 knoxnews.com .com 74
24,112 foxsports.com.au .au 75
24,141 cntraveller.com .com 81
24,190 jacksonville.com .com 74
24,233 glamourmagazine.co.uk .uk 81
24,525 zin.ru .ru 41
24,567 newspicks.com .com 44
24,593 vogue.in .in 64
24,733 humanite.fr .fr 87
24,853 nameswriter.com .com 48
24,882 uol.com .com 82
24,892 eldiariomontanes.es .es 82
24,934 svtplay.se .se 78
24,946 vogue.fr .fr 81
24,961 miaminewtimes.com .com 73
24,977 wired.it .it 83
24,983 sn.at .at 70
25,031 franceculture.fr .fr 81
25,048 tokyo-np.co.jp .jp 81
25,115 pilotonline.com .com 79
25,133 myrecipes.com .com 77
25,184 kompasiana.com .com 85
25,478 franceinter.fr .fr 80
25,839 delawareonline.com .com 74
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
26,005 people.inc .inc 64
26,114 democratandchronicle.com .com 73
26,342 straightdope.com .com 76
26,379 brisbanetimes.com.au .au 75
26,444 pap.pl .pl 39
26,474 mbta.com .com 64
26,637 brigitte.de .de 79
26,853 tallahassee.com .com 74
26,935 fd.nl .nl 73
27,113 in3x.net .net 38
27,306 stylecaster.com .com 77
27,361 westfalen-blatt.de .de 81
27,696 staradvertiser.com .com 84
27,723 geek.com .com 38
27,796 asiasociety.org .org 67
27,838 mtvuutiset.fi .fi 72
27,881 cas.org .org 89
28,021 sr.ht .ht 55
28,022 direct-book.com .com 45
28,170 suedkurier.de .de 81
28,237 zamin.uz .uz 92
28,377 menuweb.menu .menu 69
28,616 amzn.com .com 40
28,678 thenational.scot .scot 81
28,775 vogue.de .de 65
28,794 eastbaytimes.com .com 79
28,895 weatheronline.co.uk .uk 47
28,930 championcross.jp .jp 62
29,011 fraserinstitute.org .org 51
29,048 gq-magazin.de .de 66
29,139 u-cursos.cl .cl 51
29,341 geometrylitepc.io .io 70
29,390 kompas.id .id 64
29,402 mmafighting.com .com 68
29,477 hsex.icu .icu 63
29,499 conjur.com.br .br 73
29,553 hankookilbo.com .com 84
29,634 westword.com .com 77
29,639 edmontonjournal.com .com 71
29,701 thieme-connect.de .de 79
29,821 providencejournal.com .com 73
30,052 lasprovincias.es .es 82
30,074 yugipedia.com .com 58
30,196 youngchampion.jp .jp 62
30,237 clarionledger.com .com 74
30,387 bsc.es .es 37
30,443 dobrzemieszkaj.pl .pl 61
30,703 wickedlocal.com .com 75
30,712 artforum.com .com 74
30,737 diariosur.es .es 82
30,810 kentonline.co.uk .uk 76
30,986 commercialappeal.com .com 74
30,992 huffingtonpost.it .it 83
31,023 phoenixnewtimes.com .com 71
31,032 howlongtobeat.com .com 49
31,138 ideal.es .es 82
31,176 edp24.co.uk .uk 79
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,786 thespec.com .com 64
31,799 wefut.com .com 50
32,075 livedns.co.il .il 89
32,267 theargus.co.uk .uk 79
32,388 rgj.com .com 73
32,437 hoopshype.com .com 70
32,579 dallasobserver.com .com 77
32,790 tatler.com .com 82
32,803 oxfordmail.co.uk .uk 79
32,806 tvnet.lv .lv 79
32,918 laverdad.es .es 82
33,562 myporntube.com .com 86
33,793 observer.co.uk .uk 74
33,915 b254.com .com 54
34,092 portafolio.co .co 89
34,097 vogue.es .es 65
34,148 asianfanfics.com .com 58
PreviousPage 2 of 3Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Meta-ExternalFetcher reference, the full census, and the JSON endpoint.