Blocklist · Meta

Sites that block Meta-ExternalAgent

2,340 of the 29,991 domains we have measured, 8%, disallow meta-externalagent at the site root in robots.txt. Meta-ExternalAgent collects content that may be used to train a model.

Every domain here has withheld its content from Meta's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,340 of 29,991 measured domains (8%) disallow meta-externalagent at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Meta-ExternalAgent", measured 2026-11-22. https://crawlcensus.com/blocked/meta-externalagent

The list 2,340 domains

RankDomainTLDAI access score
3,869 pcworld.com .com 84
3,873 space.com .com 85
3,916 news24.com .com 56
3,918 rte.ie .ie 83
3,963 nos.nl .nl 69
3,966 bostonglobe.com .com 83
3,977 economictimes.com .com 91
4,061 screenrant.com .com 84
4,065 seekingalpha.com .com 91
4,070 ilsole24ore.com .com 78
4,079 vaticannews.va .va 44
4,084 uiowa.edu .edu 70
4,108 howtogeek.com .com 84
4,161 kompas.com .com 77
4,181 sbs.com.au .au 83
4,190 zoominfo.com .com 90
4,205 emojipedia.org .org 68
4,304 tvtropes.org .org 55
4,322 livemint.com .com 86
4,326 wiki.gg .gg 67
4,333 derstandard.at .at 56
4,334 blender.org .org 63
4,354 computerworld.com .com 83
4,407 20minutos.es .es 87
4,431 swr.de .de 76
4,484 mdr.de .de 76
4,493 wdr.de .de 86
4,512 amazon.com.tr .tr 40
4,528 skyscanner.net .net 52
4,551 yna.co.kr .kr 87
4,579 srf.ch .ch 87
4,684 msnbc.com .com 75
4,749 teacherspayteachers.com .com 69
4,753 xda-developers.com .com 83
4,791 collider.com .com 83
4,866 channel4.com .com 56
4,929 onlamp.com .com 80
4,987 dr.dk .dk 81
5,026 pcgamer.com .com 85
5,037 imageshack.com .com 71
5,068 gq.com .com 81
5,096 nine.com.au .au 71
5,146 everydayhealth.com .com 60
5,153 theage.com.au .au 75
5,158 mercurynews.com .com 78
5,205 lapresse.ca .ca 65
5,223 oilprice.com .com 90
5,229 eltiempo.com .com 85
5,268 ora.com .com 94
5,272 depositphotos.com .com 63
5,274 csmonitor.com .com 76
5,277 bookshop.org .org 74
5,293 turbify.com .com 85
5,310 kotaku.com .com 89
5,323 cbr.com .com 83
5,357 larazon.es .es 81
5,395 makeuseof.com .com 84
5,402 infobip.com .com 92
5,413 expressen.se .se 61
5,455 gamerant.com .com 81
5,457 diigo.com .com 35
5,523 architecturaldigest.com .com 83
5,547 amazon.ie .ie 40
5,678 dn.se .se 60
5,681 nationalpost.com .com 71
5,684 theepochtimes.com .com 80
5,714 francetvinfo.fr .fr 53
5,726 chefkoch.de .de 81
5,752 sankei.com .com 76
5,770 boingboing.net .net 79
5,775 tv2.no .no 70
5,786 cloudpath.live .live 44
5,788 ibtimes.com .com 84
5,834 motherjones.com .com 69
5,846 runescape.wiki .wiki 81
5,874 hs.fi .fi 68
5,876 dmitriy-mevedev-tor.top .top 53
5,879 cio.com .com 86
5,895 movieweb.com .com 84
5,912 pagesix.com .com 80
5,936 qobuz.com .com 79
6,020 giga.de .de 81
6,022 eiga.com .com 85
6,032 rtl.de .de 85
6,035 opaque.net .net 39
6,042 sporcle.com .com 87
6,062 nhk.jp .jp 64
6,064 delfi.lt .lt 83
6,089 azcentral.com .com 74
6,125 denverpost.com .com 78
6,159 instiz.net .net 58
6,180 tf1info.fr .fr 84
6,194 ifixit.com .com 73
6,228 sbnation.com .com 69
6,246 eurogamer.net .net 57
6,288 vulture.com .com 72
6,301 iata.org .org 68
6,308 wwd.com .com 78
6,335 royalroad.com .com 48
6,352 tv2.dk .dk 74
6,353 bbcgoodfood.com .com 84
6,356 freep.com .com 73
6,361 ici.fr .fr 87
6,377 gamesradar.com .com 85
6,386 huffingtonpost.co.uk .uk 82
6,391 sex-studentki.live .live 56
6,421 chavesnamao.com.br .br 80
6,512 telegram.com .com 73
6,513 sandiegouniontribune.com .com 78
6,522 eater.com .com 59
6,537 adda247.com .com 88
6,539 astro.com .com 37
6,554 osf.io .io 35
6,557 sportingnews.com .com 82
6,593 wired.co.uk .uk 64
6,609 procyclingstats.com .com 56
6,610 is.fi .fi 68
6,702 infoworld.com .com 83
6,726 rtbf.be .be 67
6,789 goo-net.com .com 90
6,793 svt.se .se 66
6,804 sportschau.de .de 85
6,808 chunichi.co.jp .jp 71
6,816 maxroll.gg .gg 59
6,817 funnyjunk.com .com 52
6,822 elcorreo.com .com 80
6,833 bergfex.at .at 81
6,895 togetter.com .com 73
6,948 nicematin.com .com 77
6,988 buzzfeednews.com .com 79
7,042 cntraveler.com .com 82
7,058 playmogo.com .com 44
7,084 spacebattles.com .com 72
7,091 kathimerini.gr .gr 68
7,106 nrc.nl .nl 84
7,109 military.com .com 84
7,118 nme.com .com 81
7,119 rp-online.de .de 79
7,127 hani.co.kr .kr 67
7,201 game8.jp .jp 72
7,206 hentaiera.com .com 57
7,247 nishinippon.co.jp .jp 69
7,250 manchestereveningnews.co.uk .uk 47
7,268 nzz.ch .ch 76
7,275 baltimoresun.com .com 76
7,294 fakt.pl .pl 78
7,349 rae.es .es 61
7,353 hentaifox.com .com 59
7,363 bhaskar.com .com 75
7,396 outsideonline.com .com 79
7,402 bitchute.com .com 42
7,423 jiji.com .com 81
7,427 nami.org .org 84
7,455 amazon.sg .sg 40
7,465 nationalgeographic.org .org 82
7,468 giallozafferano.it .it 87
7,510 indiewire.com .com 75
7,527 statnews.com .com 89
7,531 theonion.com .com 84
7,540 freeones.com .com 71
7,562 nettavisen.no .no 63
7,589 brasil247.com .com 80
7,605 minkabu.jp .jp 66
7,647 sporza.be .be 87
7,692 jetpunk.com .com 46
7,738 pitchfork.com .com 81
7,745 huffingtonpost.jp .jp 78
7,778 livehindustan.com .com 86
7,856 popsugar.com .com 75
7,863 refinery29.com .com 78
7,926 financialexpress.com .com 85
7,937 calciomercato.com .com 81
7,981 afr.com .com 72
8,030 oceanofpdf.com .com 59
8,055 gendai.media .media 83
8,094 nefisyemektarifleri.com .com 76
8,100 loudecho.ai .ai 56
8,127 my-personaltrainer.it .it 79
8,221 tubidy.llc .llc 75
8,227 sex-studentki.sale .sale 57
8,263 hochi.news .news 78
8,268 litmarket.ru .ru 87
8,270 bestjavporn.com .com 66
8,292 walesonline.co.uk .uk 47
8,299 thalesgroup.com .com 47
8,310 psychcentral.com .com 79
8,378 revistaoeste.com .com 69
8,380 veikkaus.fi .fi 67
8,385 emojicombos.com .com 47
8,412 anikototv.to .to 62
8,425 skyscanner.co.in .in 52
8,435 thecut.com .com 84
8,436 beatport.com .com 54
8,449 sextb.net .net 62
8,450 complex.com .com 91
8,455 ikman.lk .lk 81
8,484 sciencenews.org .org 86
8,511 demonicscans.org .org 57
8,560 presseportal.de .de 87
8,568 kriesi.at .at 77
PreviousPage 2 of 12Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Meta-ExternalAgent reference, the full census, and the JSON endpoint.