Blocklist · ByteDance

Sites that block Bytespider

2,817 of the 29,991 domains we have measured, 9%, disallow bytespider at the site root in robots.txt. Bytespider collects content that may be used to train a model.

Every domain here has withheld its content from ByteDance's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

ByteDance documents that this agent may ignore robots.txt, so a Disallow here records the site's stated preference rather than an enforced block.
The measurement

2,817 of 29,991 measured domains (9%) disallow bytespider at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Bytespider", measured 2026-11-22. https://crawlcensus.com/blocked/bytespider

The list 2,817 domains

RankDomainTLDAI access score
43,659 medonet.pl .pl 84
43,666 tasty.co .co 79
43,667 sweclockers.com .com 63
43,671 pbtech.co.nz .nz 82
43,715 carsdirect.com .com 86
43,742 apsnet.org .org 50
43,802 fussballdaten.de .de 76
43,883 kctv5.com .com 78
43,890 21jingji.com .com 65
43,974 ybw.com .com 87
43,983 teambelgium.net .net 59
43,986 nn.de .de 83
43,996 supermarketnews.com .com 82
44,026 gainesville.com .com 66
44,043 matters.town .town 56
44,055 270towin.com .com 72
44,057 currencyaz.com .com 55
44,095 acellus.com .com 59
44,099 wistv.com .com 76
44,116 eevblog.com .com 76
44,136 bookbeat.com .com 68
44,151 betterbanned.com .com 71
44,155 lbbonline.com .com 70
44,169 fox5vegas.com .com 76
44,183 barcablaugranes.com .com 68
44,228 notionpress.com .com 64
44,230 mcpvp.club .club 50
44,252 braunschweiger-zeitung.de .de 65
44,255 cagesideseats.com .com 72
44,310 signinapp.com .com 90
44,313 um.ac.id .id 83
44,349 nhnieuws.nl .nl 74
44,366 capitalfm.com .com 80
44,374 thedecisionlab.com .com 96
44,391 driffle.com .com 87
44,404 umm.edu .edu 77
44,418 adoptimix.com .com 68
44,440 hitmaal.io .io 71
44,489 iha.com.tr .tr 52
44,503 livepositively.com .com 72
44,521 fibre2fashion.com .com 68
44,530 parfumo.com .com 76
44,547 vavel.com .com 89
44,548 forestapp.cc .cc 97
44,556 tritan.gg .gg 61
44,575 flgov.com .com 54
44,635 wowinterface.com .com 58
44,689 notthebee.com .com 79
44,723 dailygazette.com .com 80
44,780 virgool.io .io 58
44,807 gofeminin.de .de 74
44,829 mobilism.org .org 48
44,837 winelab.ru .ru 72
44,851 z-lib.id .id 86
44,874 news-leader.com .com 62
44,878 rtvnoord.nl .nl 73
44,880 santaads.com .com 73
44,904 popup-blocker.org .org 56
44,908 mora.jp .jp 45
44,912 audima.co .co 80
44,928 wgt.com .com 80
44,930 jogatina.com .com 80
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,003 preciso.net .net 57
45,063 wkyt.com .com 78
45,064 erepublik.com .com 62
45,120 cmha.ca .ca 76
45,133 northdata.com .com 81
45,150 samplefocus.com .com 60
45,157 durev-inc.xyz .xyz 59
45,158 autoguide.com .com 87
45,159 lansingstatejournal.com .com 62
45,165 parkpage.in .in 41
45,180 pivotalweather.com .com 25
45,216 tbcpl.lol .lol 59
45,240 vogue.mx .mx 65
45,271 deckshop.pro .pro 63
45,272 rtvutrecht.nl .nl 73
45,284 athletics.app .app 65
45,298 osapublishing.org .org 65
45,299 tourradar.com .com 92
45,316 eumetsat.int .int 80
45,349 wp.de .de 65
45,372 devid.info .info 57
45,375 cleveland19.com .com 78
45,390 tupalo.com .com 57
45,440 nrz.de .de 80
45,452 landwirt.com .com 94
45,461 lgbtqnation.com .com 72
45,468 vatera.hu .hu 50
45,482 iyottube.ph .ph 68
45,509 hysky.de .de 38
45,536 bvb-alyans.ru .ru 80
45,541 classiccat.net .net 50
45,571 wz.de .de 76
45,578 chugoku-np.co.jp .jp 80
45,602 kenhub.com .com 66
45,604 beaconjournal.com .com 66
45,605 soccer365.net .net 47
45,610 iiii-tech.com .com 71
45,615 confidentialphonelookup.com .com 60
45,663 autoauctionhistory.com .com 46
45,665 nocowanie.pl .pl 75
45,680 greenvilleonline.com .com 62
45,688 bigl.ua .ua 78
45,704 diariodeleon.es .es 91
45,716 ithome.com.tw .tw 56
45,718 fanforum.com .com 47
45,771 larioja.com .com 83
45,813 maza18.me .me 73
45,828 spin.de .de 71
45,846 amazon.jp .jp 40
45,888 livre-rare-book.com .com 72
45,892 pvphq.com .com 43
45,902 fdfd.com .com 53
45,905 kob.com .com 84
45,918 revistagq.com .com 81
45,989 aftenbladet.no .no 78
46,008 alkamitech.com .com 80
46,033 waptrick.com .com 65
46,045 mod.io .io 51
46,076 bitgo.com .com 94
46,094 kibibe.com .com 35
46,104 desihub.tv .tv 82
46,106 guildlaunch.com .com 54
46,107 javxxhd.com .com 73
46,125 stuller.com .com 89
46,127 penn.museum .museum 81
46,128 svradv.com .com 46
46,134 interregeurope.eu .eu 60
46,168 bizapedia.com .com 60
46,181 gardenersworld.com .com 82
46,199 live5news.com .com 78
46,210 i3investor.com .com 53
46,217 letsrun.com .com 68
46,221 atlantanewsfirst.com .com 78
46,231 registerguard.com .com 66
46,238 travelminit.ro .ro 88
46,250 openarchieven.nl .nl 87
46,283 pcgamingwiki.com .com 70
46,290 benu.cz .cz 71
46,292 footboom1.com .com 67
46,301 redcafe.net .net 64
46,314 backpacker.com .com 78
46,339 consequenceofsound.net .net 85
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,361 ceeol.com .com 66
46,409 are-we.co.uk .uk 79
46,458 voltairenet.org .org 65
46,490 djmag.com .com 65
46,495 sowetan.co.za .za 82
46,496 veronalabs.com .com 83
46,502 adscientificindex.com .com 71
46,539 estar.jp .jp 62
46,554 bollyflix.free .free 57
46,560 raiplaysound.it .it 59
46,579 bdsmlr.com .com 41
46,583 iggm.com .com 78
46,596 soccerpunter.com .com 72
46,599 deutschlandcard.de .de 65
46,635 wikihow.life .life 76
46,638 lunapic.com .com 68
46,641 skyslope.com .com 72
46,671 edurev.in .in 89
46,691 drbokep18.com .com 71
46,713 dating.mobi .mobi 56
46,742 live-shows.net .net 53
46,748 liveresult.ru .ru 64
46,753 grfc.ru .ru 50
46,779 miarroba.com .com 68
46,802 applicantpro.com .com 66
46,805 omroepwest.nl .nl 73
46,836 unibz.it .it 72
46,844 symbl.cc .cc 71
46,849 kold.com .com 78
46,881 luluvid.com .com 51
46,923 alisweb.org .org 39
46,928 riba.org .org 59
46,949 schulferien.org .org 51
46,990 fixya.com .com 43
46,994 ruslania.com .com 62
47,029 lostarmour.info .info 51
47,039 sluurpy.com .com 89
47,065 lifehacker.jp .jp 89
47,161 pickaweb.co.uk .uk 62
47,173 eurogamer.de .de 57
47,202 parts55.com .com 51
47,206 jojowiki.com .com 83
47,243 alaphoid.com .com 57
47,259 rollingstone.de .de 87
47,281 ggjav.com .com 55
47,285 bengo4.com .com 86
47,291 accelia.net .net 67
47,299 rtvdrenthe.nl .nl 73
47,347 mafra.cz .cz 62
47,355 cmsmasters.net .net 74
47,373 hentaienvy.com .com 65
47,374 f4wonline.com .com 64
PreviousPage 13 of 15Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Bytespider reference, the full census, and the JSON endpoint.