Blocklist · Meta

Sites that block Meta-ExternalAgent

2,340 of the 29,991 domains we have measured, 8%, disallow meta-externalagent at the site root in robots.txt. Meta-ExternalAgent collects content that may be used to train a model.

Every domain here has withheld its content from Meta's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,340 of 29,991 measured domains (8%) disallow meta-externalagent at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Meta-ExternalAgent", measured 2026-11-22. https://crawlcensus.com/blocked/meta-externalagent

The list 2,340 domains

RankDomainTLDAI access score
30,986 commercialappeal.com .com 74
31,003 maxim.com .com 87
31,010 mult-porno.world .world 56
31,020 flexmls.com .com 89
31,023 phoenixnewtimes.com .com 71
31,032 howlongtobeat.com .com 49
31,057 linksbird.co .co 54
31,138 ideal.es .es 82
31,145 knowing-jesus.com .com 70
31,149 idealhome.co.uk .uk 82
31,176 edp24.co.uk .uk 79
31,189 flightconnections.com .com 49
31,195 birdlife.org .org 73
31,229 bookmeter.com .com 68
31,231 grupofpsinais.bet .bet 54
31,232 omicaso.org .org 58
31,252 forumactif.com .com 83
31,282 vndb.org .org 60
31,284 publicdomainpictures.net .net 68
31,293 mcsweeneys.net .net 78
31,320 lumendatabase.org .org 51
31,352 sportsbusinessjournal.com .com 53
31,373 yugiyu2.com .com 56
31,381 coventrytelegraph.net .net 47
31,391 xn--l3cly6b0fd0c.net .net 76
31,392 batikloreng.online .online 76
31,425 ewtnnews.com .com 70
31,434 xxxtubevideos.click .click 61
31,441 yacenetv.com .com 83
31,443 saarbruecker-zeitung.de .de 82
31,445 kingviral.baby .baby 72
31,458 cricfytvap.net .net 78
31,475 tmll7.com .com 53
31,499 bolasport.com .com 76
31,533 krishitalk.com .com 56
31,549 lohud.com .com 74
31,552 codeweavers.com .com 64
31,565 alexiskian.com .com 70
31,566 apinagadigital.com .com 64
31,571 dailyroutine.lifestyle .lifestyle 30
31,582 gamenila.com .com 56
31,591 lottosociety.com .com 75
31,593 me.game .game 36
31,598 pahindot.me .me 77
31,601 prodelibre.com.ar .ar 84
31,602 quickplayhub.in .in 56
31,623 videyli.co .co 26
31,628 xvidey.top .top 53
31,638 marieclaire.co.uk .uk 84
31,686 one-name.net .net 29
31,733 meteocast.net .net 51
31,751 e-flux.com .com 77
31,758 are.na .na 72
31,782 mastodon.online .online 49
31,786 thespec.com .com 64
31,799 wefut.com .com 50
31,850 avsforum.com .com 43
31,857 weawow.com .com 65
31,871 diariodesevilla.es .es 84
31,908 lamag.com .com 78
31,914 springerprofessional.de .de 60
31,934 comicbookmovie.com .com 78
31,986 gamemaker.io .io 72
32,032 prx.org .org 74
32,075 livedns.co.il .il 89
32,151 devonlive.com .com 47
32,169 journaldugeek.com .com 91
32,183 tellmyroute.com .com 72
32,193 carwow.co.uk .uk 93
32,201 bookriot.com .com 79
32,225 etuovi.com .com 53
32,267 theargus.co.uk .uk 79
32,279 custommapposter.com .com 77
32,345 opencv.org .org 66
32,373 gloswielkopolski.pl .pl 79
32,388 rgj.com .com 73
32,389 sepe.es .es 73
32,391 prweek.com .com 51
32,406 pissedconsumer.com .com 74
32,437 hoopshype.com .com 70
32,456 currencyrate.today .today 67
32,484 project1999.com .com 47
32,498 toptenreviews.com .com 87
32,516 newworldencyclopedia.org .org 80
32,561 skapiec.pl .pl 77
32,579 dallasobserver.com .com 77
32,580 planetadelibros.com .com 83
32,595 saveur.com .com 70
32,605 thecrag.com .com 48
32,617 flo.com.tr .tr 52
32,631 baynews9.com .com 68
32,652 focustaiwan.tw .tw 90
32,706 marianne.net .net 67
32,713 opovo.com.br .br 85
32,718 athletic.net .net 68
32,735 rsi.ch .ch 63
32,790 tatler.com .com 82
32,803 oxfordmail.co.uk .uk 79
32,806 tvnet.lv .lv 79
32,814 write.as .as 69
32,824 datasheetbank.com .com 49
32,832 newsbank.com .com 92
32,901 stocktitan.net .net 94
32,918 laverdad.es .es 82
32,919 brighttalk.com .com 68
32,921 fs.blog .blog 80
32,935 svtperformance.com .com 75
32,951 amoxibest.top .top 79
33,018 hugo-spielautomat.com .com 73
33,063 dailypost.co.uk .uk 47
33,072 tympanus.net .net 65
33,084 afi.com .com 49
33,179 srvb1.com .com 35
33,243 vgen.co .co 52
33,254 aan.com .com 74
33,285 marketresearch.com .com 72
33,323 ftdichip.com .com 75
33,325 hokkaido-np.co.jp .jp 78
33,354 pucrs.br .br 72
33,364 merca20.com .com 79
33,430 ilmanifesto.it .it 68
33,494 theanalyst.com .com 88
33,559 bmwfans.info .info 56
33,562 myporntube.com .com 86
33,568 icecat.it .it 44
33,569 nottinghampost.com .com 47
33,572 diariodolitoral.com.br .br 70
33,597 onlinesequencer.net .net 45
33,644 tv4.se .se 76
33,738 ruv.is .is 59
33,757 dealabs.com .com 78
33,793 observer.co.uk .uk 74
33,820 pieandmightymsp.com .com 68
33,845 mqseries.net .net 55
33,856 aciprensa.com .com 71
33,895 myjoyonline.com .com 88
33,915 b254.com .com 54
33,931 lancasteronline.com .com 79
33,934 carlsen.de .de 71
33,947 baby-calendar.jp .jp 76
33,953 annapurnabhandarwb.com .com 86
33,954 avxtha.com .com 74
33,961 ogomovies2.com .com 82
33,966 zonexa.xyz .xyz 73
34,012 womanandhome.com .com 85
34,023 followfollow.com .com 73
34,027 thai-novel.com .com 88
34,041 skyscanner.pl .pl 52
34,089 metopera.org .org 31
34,092 portafolio.co .co 89
34,097 vogue.es .es 65
34,148 asianfanfics.com .com 58
34,156 clickon.pro .pro 57
34,236 welib.st .st 47
34,253 indexbox.io .io 84
34,301 stevehoffman.tv .tv 65
34,309 unionleader.com .com 63
34,310 coolrom.com.au .au 62
34,372 cafemom.com .com 76
34,375 columbian.com .com 78
34,380 fitforfun.de .de 89
34,387 breakingnews.ie .ie 71
34,405 myfwc.com .com 65
34,445 monoskop.org .org 43
34,449 mylondon.news .news 47
34,491 sfweekly.com .com 74
34,528 multimovies.makeup .makeup 75
34,589 shabdkosh.com .com 83
34,603 stylist.co.uk .uk 72
34,621 ifad.org .org 69
34,636 upmsp.edu.in .in 54
34,669 allporncomics.co .co 64
34,718 sportsyou.com .com 86
34,737 vibe.com .com 80
34,750 mailplus.co.uk .uk 44
34,816 bokepking.cam .cam 81
34,822 milanofinanza.it .it 82
34,847 sydsvenskan.se .se 52
34,959 tmd.go.th .th 49
34,961 volksfreund.de .de 83
34,963 ipcamlive.com .com 60
34,969 enr.com .com 82
34,973 naplesnews.com .com 73
34,974 topagrar.com .com 85
34,986 ruhrnachrichten.de .de 75
34,991 monkeyvillage.pro .pro 35
34,995 hiphopkit.com .com 72
35,033 online-literature.com .com 47
35,062 dailypress.com .com 79
35,089 bismanonline.com .com 66
35,120 hurawatch.cx .cx 86
35,122 jiaci.org .org 52
35,158 jyllands-posten.dk .dk 70
35,165 mldkyt.com .com 50
35,198 historyextra.com .com 82
35,306 natega-eg.net .net 48
35,324 northdata.de .de 81
35,326 swr3.de .de 74
35,383 alexishave.com .com 66
35,387 alexistogel.ninja .ninja 72
PreviousPage 8 of 12Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Meta-ExternalAgent reference, the full census, and the JSON endpoint.