Blocklist · Meta

Sites that block Meta-ExternalFetcher

553 of the 29,991 domains we have measured, 2%, disallow meta-externalfetcher at the site root in robots.txt. Meta-ExternalFetcher fetches a page live when a user asks the assistant for it.

Every domain here has removed itself from Meta's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

Meta documents that this agent may ignore robots.txt, so a Disallow here records the site's stated preference rather than an enforced block.
The measurement

553 of 29,991 measured domains (2%) disallow meta-externalfetcher at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Meta-ExternalFetcher", measured 2026-11-21. https://crawlcensus.com/blocked/meta-externalfetcher

The list 553 domains

RankDomainTLDAI access score
26 amazon.com .com 40
62 msn.com .com 41
121 amazonvideo.com .com 55
260 launchpad.net .net 52
318 amazon.co.uk .uk 40
377 amazon.de .de 40
494 amazon.co.jp .jp 40
533 wired.com .com 81
571 uol.com.br .br 82
609 amazon.fr .fr 40
623 amazon.co.za .za 40
627 dailymail.co.uk .uk 61
635 amazon.ca .ca 40
659 amazon.in .in 40
662 cnet.com .com 79
681 dailymail.com .com 61
682 primevideo.com .com 71
706 amazon.it .it 40
725 amazon.es .es 40
812 amazon.com.au .au 40
815 theverge.com .com 77
823 nbcnews.com .com 59
833 amazon.com.br .br 40
926 genius.com .com 65
942 repubblica.it .it 85
983 elmundo.es .es 84
1,019 theconversation.com .com 62
1,027 huffpost.com .com 85
1,032 amazon.com.mx .mx 40
1,056 marca.com .com 85
1,073 bfmtv.com .com 81
1,083 nypost.com .com 81
1,098 franceinfo.fr .fr 60
1,177 pimpbunny.com .com 56
1,223 furaffinity.net .net 39
1,275 huffingtonpost.com .com 85
1,286 actu.fr .fr 81
1,300 f95zone.to .to 59
1,305 dw.com .com 42
1,376 dagbladet.no .no 74
1,394 usnews.com .com 83
1,409 iltalehti.fi .fi 58
1,442 buzzfeed.com .com 65
1,446 ekstrabladet.dk .dk 67
1,456 eenadu.net .net 67
1,639 darkero.com .com 67
1,793 newyorker.com .com 73
1,831 zdnet.com .com 76
1,835 wikihow.com .com 76
1,984 themoviedb.org .org 74
1,993 ign.com .com 62
2,006 amazon.eg .eg 40
2,010 mashable.com .com 61
2,084 justice.gov .gov 54
2,118 variety.com .com 80
2,325 nrk.no .no 69
2,348 nontondrama.my .my 77
2,421 entrepreneur.com .com 89
2,423 pcmag.com .com 63
2,530 smh.com.au .au 82
2,561 technologyreview.com .com 82
2,581 hollywoodreporter.com .com 79
2,691 newscientist.com .com 76
2,723 yumpu.com .com 30
2,786 chicagotribune.com .com 79
2,822 indianexpress.com .com 85
2,899 vanityfair.it .it 77
2,969 today.com .com 62
2,989 vogue.com .com 82
3,198 metro.co.uk .uk 59
3,253 billboard.com .com 78
3,286 flipboard.com .com 64
3,342 lifehacker.com .com 65
3,347 consumerreports.org .org 72
3,368 nzherald.co.nz .nz 56
3,371 nymag.com .com 74
3,418 deadline.com .com 81
3,499 amazon.se .se 40
3,520 amazon.ae .ae 40
3,533 vanityfair.com .com 81
3,576 abc.es .es 81
3,595 politico.eu .eu 62
3,598 amazon.nl .nl 40
3,916 news24.com .com 56
3,963 nos.nl .nl 69
4,161 kompas.com .com 77
4,181 sbs.com.au .au 83
4,190 zoominfo.com .com 90
4,512 amazon.com.tr .tr 40
4,592 lastampa.it .it 81
5,068 gq.com .com 81
5,096 nine.com.au .au 71
5,146 everydayhealth.com .com 60
5,153 theage.com.au .au 75
5,158 mercurynews.com .com 78
5,205 lapresse.ca .ca 65
5,229 eltiempo.com .com 85
5,413 expressen.se .se 61
5,457 diigo.com .com 35
5,523 architecturaldigest.com .com 83
5,547 amazon.ie .ie 40
5,678 dn.se .se 60
5,681 nationalpost.com .com 71
5,714 francetvinfo.fr .fr 53
5,726 chefkoch.de .de 81
5,752 sankei.com .com 76
5,775 tv2.no .no 70
5,788 ibtimes.com .com 84
5,874 hs.fi .fi 68
5,912 pagesix.com .com 80
6,089 azcentral.com .com 74
6,125 denverpost.com .com 78
6,159 instiz.net .net 58
6,180 tf1info.fr .fr 84
6,228 sbnation.com .com 69
6,246 eurogamer.net .net 57
6,288 vulture.com .com 72
6,308 wwd.com .com 78
6,352 tv2.dk .dk 74
6,356 freep.com .com 73
6,361 ici.fr .fr 87
6,386 huffingtonpost.co.uk .uk 82
6,512 telegram.com .com 73
6,513 sandiegouniontribune.com .com 78
6,522 eater.com .com 59
6,557 sportingnews.com .com 82
6,593 wired.co.uk .uk 64
6,609 procyclingstats.com .com 56
6,610 is.fi .fi 68
6,726 rtbf.be .be 67
6,793 svt.se .se 66
6,808 chunichi.co.jp .jp 71
6,816 maxroll.gg .gg 59
6,822 elcorreo.com .com 80
6,948 nicematin.com .com 77
6,988 buzzfeednews.com .com 79
7,042 cntraveler.com .com 82
7,127 hani.co.kr .kr 67
7,247 nishinippon.co.jp .jp 69
7,268 nzz.ch .ch 76
7,275 baltimoresun.com .com 76
7,363 bhaskar.com .com 75
7,423 jiji.com .com 81
7,455 amazon.sg .sg 40
7,465 nationalgeographic.org .org 82
7,510 indiewire.com .com 75
7,562 nettavisen.no .no 63
7,647 sporza.be .be 87
7,738 pitchfork.com .com 81
7,745 huffingtonpost.jp .jp 78
7,856 popsugar.com .com 75
7,863 refinery29.com .com 78
7,926 financialexpress.com .com 85
7,981 afr.com .com 72
8,030 oceanofpdf.com .com 59
8,263 hochi.news .news 78
8,380 veikkaus.fi .fi 67
8,435 thecut.com .com 84
8,449 sextb.net .net 62
8,484 sciencenews.org .org 86
8,606 javdb.com .com 52
8,789 supremevalues.com .com 70
8,799 hmetro.com.my .my 60
8,830 portalsamorzadowy.pl .pl 69
8,854 nikkeibp.co.jp .jp 64
8,883 manhuagui.com .com 61
8,897 rynekzdrowia.pl .pl 64
8,959 glamour.com .com 81
9,062 bluewin.ch .ch 66
9,069 wbur.org .org 81
9,149 forth.gr .gr 60
9,339 detroitnews.com .com 68
9,388 law.com .com 68
9,542 findnovel.net .net 64
10,191 paryajlakay.com .com 45
10,289 interestingengineering.com .com 87
10,291 usmagazine.com .com 87
10,345 inews.co.uk .uk 67
10,432 expansion.com .com 81
10,470 babycenter.com .com 62
10,495 financialpost.com .com 71
10,501 ocregister.com .com 80
10,552 self.com .com 68
10,708 vogue.co.uk .uk 79
10,781 47news.jp .jp 67
10,826 bonappetit.com .com 82
10,885 livestrong.com .com 80
10,894 augsburger-allgemeine.de .de 81
10,934 huffingtonpost.fr .fr 81
11,130 lexpress.fr .fr 86
11,201 latercera.com .com 80
11,242 teenvogue.com .com 82
11,349 posteo.de .de 67
11,390 pantone.com .com 78
11,457 wnp.pl .pl 65
11,504 gamblingtherapy.org .org 87
11,610 allure.com .com 84
11,934 loksatta.com .com 85
12,113 bostonherald.com .com 78
12,377 delfi.ee .ee 57
Page 1 of 3Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Meta-ExternalFetcher reference, the full census, and the JSON endpoint.