Blocklist · Common Crawl Foundation

Sites that block CCBot

2,881 of the 29,991 domains we have measured, 10%, disallow ccbot at the site root in robots.txt. CCBot archives content.

Every domain here has withheld its content from Common Crawl Foundation's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,881 of 29,991 measured domains (10%) disallow ccbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block CCBot", measured 2026-11-21. https://crawlcensus.com/blocked/ccbot

The list 2,881 domains

RankDomainTLDAI access score
13,667 dispatch.com .com 74
13,696 blackfaggot.online .online 42
13,719 payoneer.com .com 79
13,737 unstop.com .com 55
13,745 abczdrowie.pl .pl 92
13,772 khan.co.kr .kr 72
13,792 sheknows.com .com 83
13,825 heraldscotland.com .com 81
13,828 aeon.co .co 84
13,840 writeablog.net .net 51
13,860 francebleu.fr .fr 87
13,871 ilex.by .by 59
13,873 rappler.com .com 80
13,879 tennessean.com .com 74
13,882 vermont.gov .gov 73
13,930 myminifactory.com .com 38
13,932 bokepi.com .com 75
13,937 songkick.com .com 66
13,942 dnevnik.hr .hr 55
13,947 prevention.com .com 74
13,987 mydealz.de .de 79
14,009 abc7chicago.com .com 81
14,034 lrt.lt .lt 84
14,065 animesuge.cz .cz 71
14,079 wccftech.com .com 80
14,087 diepresse.com .com 80
14,090 simon.com .com 31
14,092 politiken.dk .dk 73
14,102 elledecor.com .com 76
14,112 project-syndicate.org .org 88
14,153 ikiru.wtf .wtf 67
14,163 car-part.com .com 49
14,168 ncaa.com .com 85
14,248 bhf.org.uk .uk 89
14,342 gothamist.com .com 69
14,350 abc7ny.com .com 81
14,374 delfi.lv .lv 80
14,404 cumhuriyet.com.tr .tr 84
14,408 nbclosangeles.com .com 33
14,468 mediapart.fr .fr 84
14,470 gamedeveloper.com .com 83
14,474 seiska.fi .fi 79
14,491 aerzteblatt.de .de 69
14,512 listennotes.com .com 64
14,553 investorgain.com .com 83
14,562 nbcchicago.com .com 62
14,576 porn-sex-site.com .com 77
14,612 mopo.de .de 79
14,626 autoscout24.com .com 70
14,661 jalalive.news .news 82
14,733 waz.de .de 82
14,774 zebzoub.com .com 58
14,790 bokepindo13.study .study 71
14,794 sexeti.com .com 59
14,805 redalyc.org .org 52
14,809 auto-motor-und-sport.de .de 81
14,826 rockpapershotgun.com .com 58
14,828 autotrader.ca .ca 81
14,833 katmoviehd.top .top 70
14,872 mz.de .de 70
14,873 rareanimes.mov .mov 72
14,878 postheaven.net .net 54
14,902 xn--3-wxfc3gwbi.net .net 77
14,903 2osb.com .com 48
14,904 kyalsin.com .com 47
14,931 cossunex.com .com 59
14,961 townnews.com .com 75
14,973 lek-manga.net .net 68
14,976 glvietsub.net .net 82
14,982 finrezo.com .com 56
14,998 kino.de .de 84
15,017 mhanational.org .org 88
15,027 desidisini.space .space 26
15,046 drbokep.top .top 71
15,056 dindadisini.space .space 26
15,061 jnedemure.com .com 69
15,064 leadsafrica.com .com 56
15,067 rapbeh.cx .cx 35
15,068 toonstream.vip .vip 56
15,076 analdin.com .com 44
15,080 adl.org .org 87
15,146 aldat.online .online 39
15,149 mochahost.com .com 72
15,174 pcgamesn.com .com 83
15,211 zenwriting.net .net 56
15,215 sportngin.com .com 84
15,218 statesman.com .com 51
15,230 mysanantonio.com .com 54
15,233 thieme-connect.com .com 39
15,260 birminghammail.co.uk .uk 47
15,269 gamesindustry.biz .biz 57
15,291 phoronix.com .com 56
15,293 abc7news.com .com 81
15,297 familie.de .de 84
15,311 mcmaster.com .com 69
15,314 europe1.fr .fr 69
15,321 coursehero.com .com 76
15,329 estrepublicain.fr .fr 82
15,336 axiombid.com .com 73
15,352 aaas.org .org 65
15,366 filmix.my .my 72
15,372 letudiant.fr .fr 81
15,391 netzwelt.de .de 84
15,396 vinted.nl .nl 86
15,401 desmoinesregister.com .com 74
15,421 mtggoldfish.com .com 63
15,458 utopia.de .de 85
15,467 betsapi.com .com 59
15,476 computerbild.de .de 81
15,486 angkanet22.com .com 63
15,488 nagawinplus.com .com 69
15,490 vidleey.sbs .sbs 43
15,491 wetheleader.org .org 65
15,493 churchcenter.com .com 58
15,508 ppy.sh .sh 59
15,509 diariovasco.com .com 82
15,511 chocolatey.org .org 61
15,557 cincinnati.com .com 74
15,559 jyu.fi .fi 77
15,570 finna.fi .fi 54
15,571 recipetineats.com .com 82
15,607 townandcountrymag.com .com 76
15,626 moviesmod.army .army 35
15,637 education.com .com 93
15,650 asd.ink .ink 79
15,668 azoramoon.com .com 73
15,671 narto-drama.com .com 41
15,679 opendota.com .com 42
15,686 postimees.ee .ee 66
15,697 xn--3-wxfc3gwbi.com .com 77
15,700 1x1x5.com .com 55
15,710 yourstory.com .com 55
15,723 disboard.org .org 52
15,743 alexistogel.link .link 66
15,744 alexistogel.mobi .mobi 74
15,745 alexistogel.news .news 76
15,746 alexistogel.red .red 76
15,747 alexistogel.team .team 78
15,748 alexistogel.tips .tips 76
15,752 kuttymovies1.fast .fast 54
15,760 vildey.space .space 43
15,782 gamesview.shop .shop 70
15,804 moviesda.bet .bet 56
15,806 tamilprint61.art .art 55
15,830 xn--3-7wf9a3c4b3bt8f0cya.net .net 77
15,834 bokepindo13.party .party 71
15,853 1tamilmv.cards .cards 74
15,856 aagmaal.bet .bet 72
15,857 aghazno11.com .com 80
15,858 alexistogel.lol .lol 73
15,867 holoo11.com .com 84
15,868 holoo12.com .com 84
15,873 mawartotogoku.com .com 74
15,878 mingky.live .live 52
15,879 mkvmoviespoint.cab .cab 72
15,882 moviesda.talk .talk 56
15,885 olxcun.com .com 75
15,888 pinayflix.app .app 51
15,890 prmovies.farm .farm 70
15,898 skymovieshd.spot .spot 53
15,900 tamilprint60.art .art 55
15,909 toto12ultra.com .com 75
15,910 toto12wind.com .com 77
15,918 yallashots.top .top 72
15,919 ydparty.tv .tv 69
15,944 rtl.be .be 82
16,013 bunte.de .de 87
16,015 blogfreely.net .net 34
16,036 palletsprojects.com .com 59
16,047 longisland.com .com 51
16,058 globalsecurity.org .org 58
16,082 t3n.de .de 78
16,086 nintendo.co.jp .jp 50
16,103 dweb.link .link 65
16,118 healio.com .com 69
16,164 christianitytoday.com .com 89
16,182 sermonaudio.com .com 71
16,191 urdupoint.com .com 68
16,209 ipfs.io .io 65
16,223 smashwords.com .com 58
16,227 deadspin.com .com 86
16,251 nysed.gov .gov 69
16,252 vivino.com .com 82
16,254 unil.ch .ch 83
16,256 runnerspace.com .com 47
16,283 positivepsychology.com .com 76
16,290 duden.de .de 71
16,317 ilmattino.it .it 62
16,323 ard.de .de 76
16,332 vinted.cz .cz 68
16,365 xozilla.com .com 58
16,377 decider.com .com 79
16,385 spokesman.com .com 75
16,421 dejure.org .org 72
16,423 iwm.org.uk .uk 77
16,428 e24.no .no 78
16,444 hometalk.com .com 63
16,473 skilljar.com .com 76
16,478 wnba.com .com 69
16,520 hotukdeals.com .com 78
PreviousPage 5 of 15Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the CCBot reference, the full census, and the JSON endpoint.