Blocklist · Meta

Sites that block Meta-ExternalAgent

2,340 of the 29,991 domains we have measured, 8%, disallow meta-externalagent at the site root in robots.txt. Meta-ExternalAgent collects content that may be used to train a model.

Every domain here has withheld its content from Meta's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,340 of 29,991 measured domains (8%) disallow meta-externalagent at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Meta-ExternalAgent", measured 2026-11-22. https://crawlcensus.com/blocked/meta-externalagent

The list 2,340 domains

RankDomainTLDAI access score
39,303 watoday.com.au .au 75
39,349 hoy.es .es 82
39,350 thehighline.org .org 77
39,369 santafenewmexican.com .com 76
39,393 zakzak.co.jp .jp 72
39,471 vwvortex.com .com 43
39,516 muz.uz .uz 70
39,519 vinepair.com .com 84
39,570 roletapixbr.com .com 51
39,587 hulldailymail.co.uk .uk 47
39,597 indifferentlanguages.com .com 52
39,657 korben.info .info 77
39,692 netdoktor.de .de 79
39,703 deepsilver.com .com 70
39,706 indiehackers.com .com 54
39,712 vsbattles.com .com 68
39,713 learner.org .org 65
39,806 mercateo.com .com 48
39,824 werstreamt.es .es 59
39,976 aifpleasurebeh.org .org 35
40,005 bobdylan.com .com 58
40,006 gemtracks.com .com 66
40,013 belfastlive.co.uk .uk 47
40,040 timeedit.net .net 53
40,049 hpcwire.com .com 63
40,102 eurasianet.org .org 56
40,114 cgm.com .com 75
40,163 worldnomads.com .com 81
40,219 thegrio.com .com 89
40,258 listchallenges.com .com 48
40,278 dmhy.org .org 59
40,317 iza.ne.jp .jp 79
40,322 neatorama.com .com 55
40,328 bntu.by .by 80
40,339 kompas.tv .tv 76
40,360 noen.at .at 79
40,367 centralcharts.com .com 58
40,392 vanguardia.com.mx .mx 84
40,403 thelocal.fr .fr 87
40,411 kmib.co.kr .kr 47
40,413 dualshockers.com .com 83
40,487 opendatabot.ua .ua 58
40,512 thetelegraphandargus.co.uk .uk 78
40,536 snl.no .no 87
40,537 sarna.net .net 81
40,631 technavio.com .com 89
40,638 main-echo.de .de 64
40,703 mrst.one .one 52
40,713 dwds.de .de 64
40,740 sheetjs.com .com 59
40,741 theblast.com .com 92
40,796 elpasotimes.com .com 73
40,805 straight.com .com 79
40,817 cancilleria.gov.co .co 60
40,830 xink.io .io 78
40,858 inspiredtaste.net .net 70
40,859 practicalmachinist.com .com 67
40,868 sayweee.com .com 72
40,874 grid.id .id 64
40,876 cntraveller.in .in 81
40,904 nesn.com .com 73
40,912 saambaa.com .com 66
40,961 wplace.live .live 73
40,968 kontan.co.id .id 77
40,969 nrn.com .com 81
40,971 amlegal.com .com 76
40,973 amainhobbies.com .com 47
41,083 summitpost.org .org 56
41,098 gqmagazine.fr .fr 82
41,104 stokesentinel.co.uk .uk 47
41,121 lancashiretelegraph.co.uk .uk 78
41,203 analdin.xxx .xxx 44
41,227 musescore.org .org 55
41,290 theweek.co.uk .uk 84
41,295 newson6.com .com 82
41,308 sunbeltrentals.com .com 79
41,310 businessinsider.jp .jp 89
41,317 waymarking.com .com 48
41,319 uwsp.edu .edu 90
41,336 archinect.com .com 68
41,349 planet-wissen.de .de 71
41,357 mamul.am .am 66
41,394 capitalizemytitle.com .com 71
41,420 prq.to .to 42
41,459 saharareporters.com .com 53
41,465 naughtyamerica.com .com 54
41,519 greenme.it .it 82
41,542 oefb.at .at 50
41,577 weblly.com .com 86
41,590 at5.nl .nl 70
41,603 lci.fr .fr 84
41,614 thebulwark.com .com 54
41,647 cjonline.com .com 73
41,652 flowmusic.app .app 64
41,653 donaukurier.de .de 78
41,678 usna.edu .edu 71
41,709 seneweb.com .com 86
41,713 roanoke.com .com 80
41,721 canarias7.es .es 83
41,736 eadt.co.uk .uk 79
41,760 dailybreeze.com .com 76
41,824 smiletemplates.com .com 47
41,826 defcon.org .org 66
41,851 pixelfed.social .social 46
41,865 thehealthy.com .com 87
41,872 vanityfair.fr .fr 66
41,892 icook.tw .tw 89
41,947 vogue.co.jp .jp 65
41,958 theledger.com .com 74
41,962 okinawatimes.co.jp .jp 79
41,971 gallimard.fr .fr 77
41,980 aqi.in .in 75
41,987 intermountainhealthcare.org .org 63
41,999 slub-dresden.de .de 71
42,010 aprs.fi .fi 41
42,076 iex.nl .nl 48
42,086 nintendoeverything.com .com 89
42,099 ephotozine.com .com 64
42,112 news9.com .com 85
42,113 glasgowtimes.co.uk .uk 79
42,160 tcpalm.com .com 73
42,192 weserv.nl .nl 55
42,222 boomi.in .in 77
42,248 architecture.com .com 59
42,260 rcgroups.com .com 41
42,262 advrider.com .com 41
42,289 warosu.org .org 47
42,293 tfd.com .com 69
42,336 eweka.nl .nl 75
42,353 priuschat.com .com 55
42,423 welib.org .org 49
42,463 grillio.com .com 53
42,505 medietall.no .no 48
42,512 tech-ties.net .net 85
42,582 decanter.com .com 85
42,592 erowid.org .org 40
42,595 101soundboards.com .com 78
42,607 wruser.org .org 43
42,611 podimo.com .com 70
42,614 citizen-times.com .com 74
42,651 computerwoche.de .de 85
42,670 wy903.com .com 53
42,682 opinautos.com .com 52
42,698 sol.no .no 74
42,800 1000kitap.com .com 42
42,835 iodonna.it .it 86
42,842 bahamaslocal.com .com 61
42,849 hoganlovells.com .com 81
42,862 horseracingnation.com .com 72
42,863 textstudio.com .com 74
42,874 kone.gg .gg 58
42,906 bilh.org .org 92
42,919 tcrf.net .net 29
42,934 v2006.com .com 49
42,960 ceskenoviny.cz .cz 68
42,981 viewbug.com .com 83
43,055 deepdreamgenerator.com .com 79
43,117 polycasino.com .com 76
43,126 xfolio.jp .jp 72
43,132 jtbc.co.kr .kr 70
43,207 tv4play.se .se 77
43,218 sermoncentral.com .com 59
43,251 sufficientvelocity.com .com 73
43,268 candlepowerforums.com .com 61
43,281 xvix.eu .eu 75
43,302 seacoastonline.com .com 69
43,309 tonkeeper.com .com 57
43,315 stateline.org .org 69
43,329 skinnytaste.com .com 90
43,346 aerztezeitung.de .de 90
43,414 cameradb.review .review 62
43,415 pokeheroes.com .com 57
43,437 derbytelegraph.co.uk .uk 47
43,455 noi.md .md 54
43,480 remoteok.com .com 76
43,544 nautiljon.com .com 62
43,587 admagazine.fr .fr 81
43,605 zonerama.com .com 30
43,659 medonet.pl .pl 84
43,666 tasty.co .co 79
43,667 sweclockers.com .com 63
43,715 carsdirect.com .com 86
43,742 apsnet.org .org 50
43,802 fussballdaten.de .de 76
43,890 21jingji.com .com 65
43,909 gipuzkoa.eus .eus 70
43,974 ybw.com .com 87
43,983 teambelgium.net .net 59
43,986 nn.de .de 83
43,996 supermarketnews.com .com 82
44,026 gainesville.com .com 66
44,041 cambridge-news.co.uk .uk 47
44,043 matters.town .town 56
44,055 270towin.com .com 72
44,057 currencyaz.com .com 55
44,116 eevblog.com .com 76
44,136 bookbeat.com .com 68
44,151 betterbanned.com .com 71
44,155 lbbonline.com .com 70
44,183 barcablaugranes.com .com 68
PreviousPage 10 of 12Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Meta-ExternalAgent reference, the full census, and the JSON endpoint.