Blocklist · ByteDance

Sites that block Bytespider

2,817 of the 29,991 domains we have measured, 9%, disallow bytespider at the site root in robots.txt. Bytespider collects content that may be used to train a model.

Every domain here has withheld its content from ByteDance's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

ByteDance documents that this agent may ignore robots.txt, so a Disallow here records the site's stated preference rather than an enforced block.
The measurement

2,817 of 29,991 measured domains (9%) disallow bytespider at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Bytespider", measured 2026-11-22. https://crawlcensus.com/blocked/bytespider

The list 2,817 domains

RankDomainTLDAI access score
6,804 sportschau.de .de 85
6,808 chunichi.co.jp .jp 71
6,816 maxroll.gg .gg 59
6,817 funnyjunk.com .com 52
6,820 money.pl .pl 89
6,822 elcorreo.com .com 80
6,824 vidyard.com .com 97
6,865 illinois.gov .gov 67
6,895 togetter.com .com 73
6,922 livescore.in .in 71
6,934 blesk.cz .cz 62
6,945 countryliving.com .com 74
6,948 nicematin.com .com 77
6,988 buzzfeednews.com .com 79
7,042 cntraveler.com .com 82
7,058 playmogo.com .com 44
7,082 aktuality.sk .sk 85
7,084 spacebattles.com .com 72
7,091 kathimerini.gr .gr 68
7,106 nrc.nl .nl 84
7,109 military.com .com 84
7,118 nme.com .com 81
7,119 rp-online.de .de 79
7,127 hani.co.kr .kr 67
7,177 nypl.org .org 46
7,178 sanspo.com .com 77
7,190 espn.in .in 48
7,201 game8.jp .jp 72
7,206 hentaiera.com .com 57
7,225 ygosu.com .com 64
7,239 gala.de .de 76
7,247 nishinippon.co.jp .jp 69
7,268 nzz.ch .ch 76
7,275 baltimoresun.com .com 76
7,294 fakt.pl .pl 78
7,324 nordkurier.de .de 69
7,334 flightrising.com .com 62
7,345 rpp.pe .pe 90
7,349 rae.es .es 61
7,353 hentaifox.com .com 59
7,355 bandlab.com .com 59
7,360 mathrubhumi.com .com 82
7,363 bhaskar.com .com 75
7,372 15min.lt .lt 74
7,392 delish.com .com 76
7,396 outsideonline.com .com 79
7,402 bitchute.com .com 42
7,423 jiji.com .com 81
7,427 nami.org .org 84
7,432 nouvelobs.com .com 87
7,437 espn.com.br .br 48
7,455 amazon.sg .sg 40
7,465 nationalgeographic.org .org 82
7,468 giallozafferano.it .it 87
7,489 ilrestodelcarlino.it .it 81
7,510 indiewire.com .com 75
7,514 biography.com .com 76
7,527 statnews.com .com 89
7,531 theonion.com .com 84
7,540 freeones.com .com 71
7,556 lindependant.fr .fr 86
7,562 nettavisen.no .no 63
7,589 brasil247.com .com 80
7,605 minkabu.jp .jp 66
7,647 sporza.be .be 87
7,680 wordplays.com .com 56
7,692 jetpunk.com .com 46
7,732 rolimons.com .com 83
7,738 pitchfork.com .com 81
7,740 ridibooks.com .com 79
7,745 huffingtonpost.jp .jp 78
7,791 pixai.art .art 94
7,856 popsugar.com .com 75
7,863 refinery29.com .com 78
7,901 airbus.com .com 47
7,922 washingtonexaminer.com .com 83
7,926 financialexpress.com .com 85
7,937 calciomercato.com .com 81
7,947 cnnamador.com .com 74
7,981 afr.com .com 72
8,030 oceanofpdf.com .com 59
8,054 bebee.com .com 87
8,055 gendai.media .media 83
8,093 sbicard.com .com 73
8,094 nefisyemektarifleri.com .com 76
8,100 loudecho.ai .ai 56
8,104 x-x-x.tube .tube 84
8,127 my-personaltrainer.it .it 79
8,144 myheritage.com .com 68
8,160 javhdporn.net .net 71
8,173 leggo.it .it 62
8,207 flashscore.pl .pl 71
8,221 tubidy.llc .llc 75
8,227 sex-studentki.sale .sale 57
8,263 hochi.news .news 78
8,268 litmarket.ru .ru 87
8,270 bestjavporn.com .com 66
8,283 americatv.com.pe .pe 89
8,287 javdock.com .com 65
8,310 psychcentral.com .com 79
8,364 eredmenyek.com .com 71
8,378 revistaoeste.com .com 69
8,380 veikkaus.fi .fi 67
8,385 emojicombos.com .com 47
8,398 livesport.cz .cz 71
8,412 anikototv.to .to 62
8,435 thecut.com .com 84
8,436 beatport.com .com 54
8,439 espn.com.ar .ar 48
8,449 sextb.net .net 62
8,450 complex.com .com 91
8,484 sciencenews.org .org 86
8,511 demonicscans.org .org 57
8,541 91porna.com .com 88
8,560 presseportal.de .de 87
8,568 kriesi.at .at 77
8,599 soccerway.com .com 73
8,604 eneba.com .com 88
8,606 javdb.com .com 52
8,621 momvids.com .com 50
8,647 p-world.co.jp .jp 69
8,664 voetbalzone.nl .nl 81
8,675 typingtest.com .com 86
8,692 olympusxyz.com .com 56
8,710 rezultati.com .com 71
8,773 epey.com .com 52
8,789 supremevalues.com .com 70
8,799 hmetro.com.my .my 60
8,800 sefan.ru .ru 43
8,803 libramemoria.com .com 91
8,815 olimpbet.kz .kz 44
8,830 portalsamorzadowy.pl .pl 69
8,854 nikkeibp.co.jp .jp 64
8,866 theweek.com .com 84
8,883 manhuagui.com .com 61
8,897 rynekzdrowia.pl .pl 64
8,899 flashscore.co.id .id 70
8,928 nottingham.ac.uk .uk 64
8,931 camel1.tv .tv 67
8,944 pomorska.pl .pl 77
8,959 glamour.com .com 81
8,984 getmodsapk.com .com 75
9,007 flashscore.co .co 69
9,017 hentai20.io .io 69
9,051 avjb.com .com 53
9,055 porn-comic.com .com 54
9,062 bluewin.ch .ch 66
9,069 wbur.org .org 81
9,070 towardsdatascience.com .com 70
9,072 flashscore.ua .ua 70
9,094 flashscore.ro .ro 70
9,118 bangumi.org .org 60
9,130 2ch2.net .net 43
9,149 forth.gr .gr 60
9,171 tubidy.com .com 96
9,191 ildolomiti.it .it 90
9,196 flashscore.es .es 69
9,200 nekopoi.care .care 56
9,238 xxxmovies.life .life 64
9,242 zonatmo.org .org 63
9,264 aflamk1.net .net 57
9,277 hsex.tv .tv 53
9,278 dorama.land .land 77
9,285 pkok11.com .com 52
9,302 macworld.com .com 85
9,310 komikindo.ch .ch 77
9,317 flashscore.com.ve .ve 69
9,320 flashscore.pe .pe 70
9,339 detroitnews.com .com 68
9,350 cv666.net .net 51
9,354 samehadaku.how .how 48
9,370 gogoroyal.com .com 71
9,388 law.com .com 68
9,443 q10.com .com 82
9,447 loteriasdominicanas.com .com 49
9,492 freevideo.cz .cz 48
9,542 findnovel.net .net 64
9,565 nu.or.id .id 79
9,619 modaselvim.com .com 62
9,665 yakkun.com .com 68
9,676 leaktube.net .net 72
9,677 aagag.com .com 47
9,692 forogore.net .net 55
9,701 tamilgun.now .now 70
9,713 yupmanga.com .com 69
9,726 cdnfonts.com .com 65
9,769 freedot.info .info 40
9,787 futbol11.com .com 59
9,817 naijared.com .com 68
9,847 phimsexsuong3x.net .net 73
9,856 arbpay.me .me 41
9,885 wral.com .com 87
9,903 mk.co.kr .kr 82
9,907 erotikchat4free.org .org 37
9,925 iconify.design .design 55
9,935 tvguia.es .es 74
9,946 xn--69-6tia3cb.net .net 80
9,948 xosothantai.mobi .mobi 75
9,981 xn--l3caaa5iqc3a.com .com 81
9,983 xn--72caa3cygoc0d9c.com .com 60
PreviousPage 3 of 15Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Bytespider reference, the full census, and the JSON endpoint.