Blocklist · ByteDance

Sites that block Bytespider

2,817 of the 29,991 domains we have measured, 9%, disallow bytespider at the site root in robots.txt. Bytespider collects content that may be used to train a model.

Every domain here has withheld its content from ByteDance's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

ByteDance documents that this agent may ignore robots.txt, so a Disallow here records the site's stated preference rather than an enforced block.
The measurement

2,817 of 29,991 measured domains (9%) disallow bytespider at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Bytespider", measured 2026-11-22. https://crawlcensus.com/blocked/bytespider

The list 2,817 domains

RankDomainTLDAI access score
47,393 networkofcare.org .org 53
47,424 harpersbazaar.de .de 89
47,428 wealthmanagement.com .com 81
47,455 folhavitoria.com.br .br 78
47,492 kittl.com .com 91
47,512 bestlawyers.com .com 89
47,516 novelcool.com .com 57
47,526 allgaeuer-zeitung.de .de 81
47,536 alaskasnewssource.com .com 78
47,546 thequietus.com .com 78
47,604 gazetaexpress.com .com 68
47,624 lacucinaitaliana.it .it 81
47,659 nationalgeographic.de .de 82
47,685 electrive.com .com 79
47,703 play3.de .de 64
47,709 fomos.kr .kr 51
47,741 salzburg24.at .at 85
47,796 topekalibraries.info .info 46
47,860 gpb.org .org 81
47,879 onefivenine.com .com 46
47,884 vcstar.com .com 62
47,909 moneyweek.com .com 86
47,928 apk.dog .dog 75
47,930 diariodecadiz.es .es 67
47,938 pornicom.com .com 73
47,948 theafricareport.com .com 55
47,969 sunjournal.com .com 72
48,004 ukclimbing.com .com 67
48,010 lfpress.com .com 87
48,025 museum.ru .ru 73
48,046 trinidadexpress.com .com 74
48,057 eastasiaforum.org .org 70
48,074 grubstreet.com .com 81
48,084 jurnalul.ro .ro 82
48,129 komputerswiat.pl .pl 82
48,132 awesomebooks.com .com 68
48,199 oferteo.pl .pl 82
48,222 taiwanus.net .net 51
48,235 first-server.net .net 63
48,246 kushub.net .net 26
48,298 pokebeach.com .com 66
48,352 denikn.cz .cz 75
48,380 wsls.com .com 85
48,386 charleskeith.com .com 88
48,393 lakeland.co.uk .uk 90
48,404 lesoleil.com .com 69
48,418 fireant.vn .vn 48
48,444 vogue.com.tw .tw 81
48,446 dohaxe.com .com 37
48,485 danatoto91.com .com 72
48,515 hurawatch.sx .sx 86
48,528 dorar.net .net 58
48,539 smsbower.app .app 76
48,550 argusleader.com .com 62
48,580 highradius.com .com 89
48,585 fairview.org .org 68
48,595 muragon.com .com 85
48,613 theoutdoorstrader.com .com 52
48,637 olx.com.lb .lb 75
48,667 ozsuper.com .com 36
48,685 stylemepretty.com .com 92
48,692 fakty.com.ua .ua 93
48,724 one.co.il .il 83
48,777 mtv.fi .fi 76
48,782 bokepindo13.food .food 66
48,832 anoboy.xyz .xyz 78
48,833 alo-tech.com .com 77
48,853 vosgesmatin.fr .fr 82
48,858 otz.de .de 65
48,859 castro.fm .fm 67
48,876 chiangmaitourcenter.com .com 63
48,885 ventronchip.com .com 62
48,933 trulieve.com .com 85
48,949 unison.org.uk .uk 64
49,019 diyaudio.com .com 37
49,029 experts-exchange.com .com 68
49,037 bertrand.pt .pt 65
49,059 citypopulation.de .de 54
49,061 posfie.com .com 73
49,065 ingramcontent.com .com 69
49,067 gqitalia.it .it 64
49,124 infolibre.es .es 86
49,126 scientific.net .net 52
49,147 esentire.services .services 96
49,185 grin.co .co 95
49,192 therichest.com .com 84
49,194 mailonsunday.co.uk .uk 61
49,203 wikiroutes.info .info 73
49,209 slowfood.com .com 67
49,213 tarot.com .com 71
49,245 startmycar.com .com 51
49,253 communityimpact.com .com 86
49,271 eltern.de .de 72
49,279 presstelegram.com .com 80
49,286 upmedia.mg .mg 55
49,326 supercook.com .com 40
49,343 hd.se .se 51
49,363 mmganalytics.com .com 41
49,420 gds.it .it 77
49,434 damoang.net .net 80
49,447 businessinsider.es .es 87
49,449 cinehd.app .app 62
49,464 ucc.edu.gh .gh 94
49,474 wagwalking.com .com 95
49,495 latoto91.com .com 74
49,499 immovlan.be .be 59
49,501 sj-r.com .com 64
49,502 nagawinid.com .com 71
49,512 nzb.life .life 76
49,526 wowt.com .com 78
49,574 montgomeryadvertiser.com .com 62
49,575 stress.org .org 70
49,581 idnfinancials.com .com 75
49,592 latintimes.com .com 72
49,611 gamedesire.com .com 85
49,684 fox8live.com .com 78
49,688 olxsop.com .com 61
49,695 pricesmart.com .com 61
49,706 faymas.in .in 81
49,713 retail.ru .ru 89
49,766 jigidi.com .com 71
49,784 wdbj7.com .com 76
49,802 pazar3.mk .mk 73
49,805 gazetakrakowska.pl .pl 79
49,837 magma.com .com 58
49,879 fandomwire.com .com 67
49,883 transperfect.com .com 55
49,895 butenunbinnen.de .de 81
49,903 beautifulhalo.com .com 58
49,908 hvacdirect.com .com 81
49,925 aamulehti.fi .fi 66
50,001 toon-stream.site .site 56
50,003 europeanfilmgateway.eu .eu 65
50,021 streamate.com .com 54
50,032 collegedekho.com .com 87
50,054 tekstowo.pl .pl 74
50,063 textileexchange.org .org 70
50,070 retrogames.cz .cz 60
50,085 selectra.info .info 90
50,125 echo24.cz .cz 69
50,141 umh.es .es 68
50,168 sabq.org .org 72
50,172 palworld-lab.com .com 65
50,173 andscape.com .com 86
50,180 banglanews24.com .com 72
50,185 miricanvas.com .com 89
50,210 rummyapklist.com .com 78
50,215 newscast.jp .jp 87
50,233 thefenomeno.com .com 74
50,240 moviezmad.cool .cool 71
50,256 nipic.com .com 44
50,278 burlingtonfreepress.com .com 62
50,312 mujerhoy.com .com 80
50,314 thesportzfyapps.com .com 83
50,338 ohtuleht.ee .ee 65
50,342 thenewslens.com .com 71
50,359 verticalresponse.com .com 92
50,367 blipfoto.com .com 57
50,384 spareroom.com .com 77
50,394 ardaudiothek.de .de 85
50,417 vidoyyo.space .space 26
50,418 gagadget.com .com 72
50,452 natureetdecouvertes.com .com 84
50,461 gamioo5.com .com 56
50,478 scryfall.io .io 68
50,487 tamilchristiansongs.in .in 67
50,492 financialfilings.com .com 94
50,520 womansworld.com .com 85
50,548 fakenamegenerator.com .com 71
50,579 cleopatraslot.org .org 69
50,598 dailycal.org .org 74
50,667 charityjob.co.uk .uk 82
50,682 commonsense.org .org 86
50,684 gq.com.mx .mx 72
50,694 gabia.com .com 80
50,718 frdic.com .com 49
50,732 vesty.co.il .il 91
50,738 grantland.com .com 57
50,761 adzuna.fr .fr 51
50,803 partycity.com .com 87
50,826 yesform.com .com 84
50,854 rtcg.me .me 61
50,869 xn--4-wxfc3gwbi.com .com 77
50,877 movieinsider.com .com 70
50,891 damconuong.store .store 82
50,905 notice-facile.com .com 80
50,916 motorcycle.com .com 83
50,919 japantravel.com .com 79
50,933 nwaonline.com .com 73
50,946 dutasexx.hair .hair 66
50,953 alaskantf.com .com 45
50,960 hot.com .com 67
50,979 lopinion.fr .fr 81
50,982 nrtool.st .st 54
51,003 kenyans.co.ke .ke 78
51,009 treasure-maps.com .com 51
51,018 everydaycalculation.com .com 52
- 0daymusic.org .org 52
- aitextcleaner.pro .pro 96
- asteroidsathome.net .net 64
PreviousPage 14 of 15Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Bytespider reference, the full census, and the JSON endpoint.