Blocklist · Meta

Sites that block Meta-ExternalAgent

2,340 of the 29,991 domains we have measured, 8%, disallow meta-externalagent at the site root in robots.txt. Meta-ExternalAgent collects content that may be used to train a model.

Every domain here has withheld its content from Meta's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,340 of 29,991 measured domains (8%) disallow meta-externalagent at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Meta-ExternalAgent", measured 2026-11-22. https://crawlcensus.com/blocked/meta-externalagent

The list 2,340 domains

RankDomainTLDAI access score
17,255 thegamer.com .com 83
17,266 letemps.ch .ch 74
17,285 nextech.com .com 78
17,303 businessoffashion.com .com 68
17,420 jsbin.com .com 53
17,432 kyoto-np.co.jp .jp 82
17,455 twincities.com .com 81
17,469 palmbeachpost.com .com 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,554 abc13.com .com 79
17,596 iisd.org .org 59
17,607 philpapers.org .org 65
17,619 nst.com.my .my 42
17,634 dice.fm .fm 65
17,671 iafd.com .com 45
17,673 demilked.com .com 64
17,694 manager-magazin.de .de 63
17,721 cnn.co.jp .jp 65
17,750 6abc.com .com 79
17,785 trakt.tv .tv 58
17,796 list.ly .ly 52
17,803 gizmodo.jp .jp 88
17,833 novelphoenix.com .com 61
17,877 browser-update.org .org 57
17,886 agrarheute.com .com 75
17,889 timesofindia.com .com 72
17,914 jofogas.hu .hu 80
17,936 cinemablend.com .com 87
17,983 brightlocal.com .com 90
17,988 timeslive.co.za .za 80
17,989 clickondetroit.com .com 86
17,993 cawsg.com .com 55
18,011 genealogieonline.nl .nl 70
18,145 thewordfinder.com .com 67
18,147 volksstimme.de .de 68
18,218 globed.dev .dev 51
18,228 metafilter.com .com 51
18,266 thedirecthor.com .com 57
18,281 format.com .com 91
18,308 preyproject.com .com 93
18,309 mans.io .io 60
18,370 jamieoliver.com .com 84
18,373 instyle.de .de 89
18,402 xueqiu.com .com 47
18,442 empireonline.com .com 62
18,444 ext.to .to 57
18,490 click2houston.com .com 85
18,514 radaris.com .com 58
18,515 govtech.com .com 87
18,526 researchsquare.com .com 50
18,549 gamasutra.com .com 83
18,595 trycloudflare.com .com 52
18,607 escardio.org .org 76
18,655 capcom.com .com 59
18,687 romatoday.it .it 67
18,710 cftc.gov .gov 54
18,782 pepper.pl .pl 76
18,783 spanishdict.com .com 81
18,836 crowdworks.jp .jp 58
18,859 huntshowdown.com .com 60
18,862 namespro.ca .ca 67
18,863 dailynews.com .com 76
18,867 adressa.no .no 66
18,870 scotusblog.com .com 83
18,883 courier-journal.com .com 74
18,894 courrierinternational.com .com 83
18,896 ibs.it .it 61
18,913 armani.com .com 87
18,948 skyscanner.de .de 52
18,964 theagenticx.ai .ai 71
18,973 fictionzone.net .net 69
19,014 lovart.ai .ai 51
19,055 aftership.com .com 78
19,060 internetsociety.org .org 75
19,063 opensecrets.org .org 46
19,131 computerbase.de .de 89
19,181 comparitech.com .com 88
19,182 clickorlando.com .com 85
19,206 pseb.ac.in .in 54
19,212 sex-studentki.studio .studio 57
19,250 beatstars.com .com 51
19,278 laowang.vip .vip 53
19,283 denik.cz .cz 77
19,291 gifyu.com .com 54
19,312 pinayflix.uk .uk 46
19,314 milanotoday.it .it 67
19,342 bold.dk .dk 58
19,429 thenewstack.io .io 69
19,461 shamela.ws .ws 55
19,478 french-stream.one .one 70
19,538 xn--l3ca4b6dsc1b.com .com 77
19,550 skyscanner.fr .fr 52
19,563 blu-ray.com .com 69
19,564 playbill.com .com 68
19,576 moviesmod.at .at 77
19,614 daaz.com .com 43
19,624 fukuishimbun.co.jp .jp 76
19,631 spectrumlocalnews.com .com 45
19,632 coolrom.com .com 61
19,640 rssing.com .com 29
19,662 aagmaal.kiwi .kiwi 69
19,672 wallhaven.cc .cc 45
19,678 skymovieshd.ceo .ceo 53
19,690 equidia.fr .fr 65
19,696 givemesport.com .com 83
19,719 moviesda.blue .blue 56
19,738 echodnia.eu .eu 79
19,755 100crorescompany.com .com 40
19,762 desixxx.blog .blog 65
19,810 boutique1st.com .com 56
19,814 parfumo.de .de 74
19,836 nezavisne.com .com 69
19,839 zalora.co.id .id 89
19,860 zawya.com .com 80
19,861 meteo.gr .gr 76
19,870 gazetawroclawska.pl .pl 79
19,891 musicradar.com .com 87
19,902 kuttymovies.red .red 52
19,906 artificialintelligenceact.eu .eu 62
19,918 mkvmoviespoint.uno .uno 72
19,924 jansatta.com .com 86
19,941 ottawacitizen.com .com 71
19,957 9xflix.gdn .gdn 80
19,979 maxwinexch99.com .com 27
19,989 uyeshare.cc .cc 64
20,030 badische-zeitung.de .de 56
20,033 tradera.com .com 75
20,035 plejada.pl .pl 83
20,037 flixtor.to .to 45
20,051 sinonim.org .org 59
20,059 skyscanner.com.br .br 52
20,080 chroniclelive.co.uk .uk 47
20,152 au.int .int 63
20,166 skift.com .com 82
20,225 niemanlab.org .org 51
20,232 uv.mx .mx 75
20,270 citymapper.com .com 73
20,281 3dporndude.com .com 60
20,282 chileautos.cl .cl 68
20,310 cna.com.tw .tw 89
20,318 supersport.hr .hr 51
20,332 simplywall.st .st 95
20,337 onelook.com .com 51
20,356 fmoviess.org .org 67
20,362 thedodo.com .com 70
20,419 01net.com .com 93
20,423 watchporn.to .to 90
20,556 bravotv.com .com 74
20,575 artnews.com .com 80
20,580 daryo.uz .uz 76
20,644 artmajeur.com .com 83
20,676 haufe.de .de 76
20,718 itpro.com .com 83
20,761 nlc.hu .hu 85
20,764 nto.pl .pl 79
20,825 newspulsenow.net .net 49
20,843 tsumanne.net .net 56
20,847 gardeningknowhow.com .com 83
20,871 lavenir.net .net 83
20,888 alibris.com .com 72
20,958 pressdemocrat.com .com 80
20,964 betsfan.com .com 59
21,004 femina.hu .hu 82
21,021 cloudimg.io .io 89
21,038 mainpost.de .de 80
21,044 bps.org.uk .uk 66
21,139 ledevoir.com .com 70
21,161 telemagazyn.pl .pl 81
21,179 odibets.com .com 66
21,182 desiporn.one .one 52
21,192 technolife.com .com 90
21,231 epochtimes.com .com 85
21,248 happystove.com .com 58
21,263 pornobengala.com .com 72
21,267 seher.no .no 70
21,388 indiaxnxxx.com .com 67
21,428 plovdiv24.bg .bg 80
21,430 gamefa.com .com 95
21,431 thelocal.se .se 87
21,449 thisamericanlife.org .org 58
21,481 futureplc.com .com 86
21,504 designspiration.com .com 48
21,566 whowhatwear.com .com 82
21,586 madison.com .com 80
21,602 becekku.com .com 81
21,678 hdmaal.la .la 55
21,685 calgaryherald.com .com 71
21,692 wikidex.net .net 74
21,704 codeigniter.com .com 43
21,706 1x-chickenroad.com .com 44
21,707 khronos.org .org 68
21,736 slate.fr .fr 68
21,786 tamilprint62.art .art 55
21,792 mysiswaplace.my .my 52
21,809 cyclingnews.com .com 85
21,816 footba11.co .co 61
21,823 desy.de .de 62
21,828 wallpaper.com .com 85
21,874 jadlog.com.br .br 62
PreviousPage 5 of 12Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Meta-ExternalAgent reference, the full census, and the JSON endpoint.