Blocklist · Common Crawl Foundation

Sites that block CCBot

2,881 of the 29,991 domains we have measured, 10%, disallow ccbot at the site root in robots.txt. CCBot archives content.

Every domain here has withheld its content from Common Crawl Foundation's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,881 of 29,991 measured domains (10%) disallow ccbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block CCBot", measured 2026-11-21. https://crawlcensus.com/blocked/ccbot

The list 2,881 domains

RankDomainTLDAI access score
16,556 tourvisor.ru .ru 91
16,564 mapgenie.io .io 59
16,607 mediapost.com .com 45
16,613 goodmorningamerica.com .com 70
16,664 lightningmaps.org .org 51
16,672 pressherald.com .com 76
16,738 cnbctv18.com .com 84
16,765 scimagojr.com .com 57
16,783 taipeitimes.com .com 66
16,795 vozpopuli.com .com 87
16,796 postandcourier.com .com 77
16,803 buffalonews.com .com 80
16,814 whattoexpect.com .com 65
16,832 nltimes.nl .nl 73
16,846 porofessor.gg .gg 55
16,854 money.com .com 91
16,897 nbcbayarea.com .com 62
16,903 adn.com .com 79
16,972 nbcwashington.com .com 60
16,978 wetter.de .de 90
16,991 u.gg .gg 42
17,034 zenchef.com .com 90
17,039 robbreport.com .com 64
17,042 northjersey.com .com 74
17,055 official.link .link 72
17,081 ilfoglio.it .it 85
17,116 jn.pt .pt 66
17,131 lrb.co.uk .uk 94
17,170 wwnorton.com .com 56
17,183 semafor.com .com 72
17,189 heavyfetish.com .com 62
17,224 harpers.org .org 79
17,243 komonews.com .com 73
17,255 thegamer.com .com 83
17,266 letemps.ch .ch 74
17,285 nextech.com .com 78
17,303 businessoffashion.com .com 68
17,404 poipiku.com .com 71
17,407 placeit.net .net 68
17,420 jsbin.com .com 53
17,432 kyoto-np.co.jp .jp 82
17,455 twincities.com .com 81
17,469 palmbeachpost.com .com 73
17,483 diariodenavarra.es .es 89
17,489 dukeupress.edu .edu 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,536 seattlepi.com .com 54
17,554 abc13.com .com 79
17,561 mothercare.co.id .id 55
17,596 iisd.org .org 59
17,607 philpapers.org .org 65
17,611 komoot.de .de 72
17,619 nst.com.my .my 42
17,634 dice.fm .fm 65
17,671 iafd.com .com 45
17,673 demilked.com .com 64
17,683 capital.gr .gr 83
17,694 manager-magazin.de .de 63
17,721 cnn.co.jp .jp 65
17,724 qovani.com .com 93
17,737 audiojungle.net .net 83
17,750 6abc.com .com 79
17,785 trakt.tv .tv 58
17,796 list.ly .ly 52
17,803 gizmodo.jp .jp 88
17,833 novelphoenix.com .com 61
17,846 autoscout24.be .be 54
17,877 browser-update.org .org 57
17,886 agrarheute.com .com 75
17,914 jofogas.hu .hu 80
17,983 brightlocal.com .com 90
17,988 timeslive.co.za .za 80
17,989 clickondetroit.com .com 86
17,993 cawsg.com .com 55
17,997 iacr.org .org 58
18,011 genealogieonline.nl .nl 70
18,019 capital.fr .fr 95
18,147 volksstimme.de .de 68
18,198 489ban.net .net 90
18,214 xxxbunker.com .com 81
18,218 globed.dev .dev 51
18,228 metafilter.com .com 51
18,233 frandroid.com .com 90
18,266 thedirecthor.com .com 57
18,280 hoerzu.de .de 45
18,286 gasbuddy.com .com 67
18,308 preyproject.com .com 93
18,309 mans.io .io 60
18,311 earthsky.org .org 85
18,338 telemundo.com .com 83
18,370 jamieoliver.com .com 84
18,373 instyle.de .de 89
18,388 freemusicarchive.org .org 65
18,402 xueqiu.com .com 47
18,442 empireonline.com .com 62
18,444 ext.to .to 57
18,447 italki.com .com 93
18,473 magic.ly .ly 85
18,487 isaca.org .org 73
18,489 elcolombiano.com .com 83
18,490 click2houston.com .com 85
18,514 radaris.com .com 58
18,515 govtech.com .com 87
18,526 researchsquare.com .com 50
18,549 gamasutra.com .com 83
18,575 nanoreview.net .net 60
18,595 trycloudflare.com .com 52
18,653 roland.com .com 71
18,655 capcom.com .com 59
18,666 nicepage.com .com 89
18,687 romatoday.it .it 67
18,710 cftc.gov .gov 54
18,782 pepper.pl .pl 76
18,783 spanishdict.com .com 81
18,812 nbcphiladelphia.com .com 62
18,822 nbcdfw.com .com 33
18,859 huntshowdown.com .com 60
18,863 dailynews.com .com 76
18,867 adressa.no .no 66
18,870 scotusblog.com .com 83
18,883 courier-journal.com .com 74
18,894 courrierinternational.com .com 83
18,939 qgis.org .org 74
18,964 theagenticx.ai .ai 71
18,973 fictionzone.net .net 69
18,978 dexerto.com .com 78
18,992 tokmanni.fi .fi 85
19,014 lovart.ai .ai 51
19,060 internetsociety.org .org 75
19,063 opensecrets.org .org 46
19,181 comparitech.com .com 88
19,182 clickorlando.com .com 85
19,206 pseb.ac.in .in 54
19,212 sex-studentki.studio .studio 57
19,229 wjla.com .com 71
19,250 beatstars.com .com 51
19,278 laowang.vip .vip 53
19,283 denik.cz .cz 77
19,291 gifyu.com .com 54
19,298 nbcmiami.com .com 60
19,312 pinayflix.uk .uk 46
19,314 milanotoday.it .it 67
19,322 oprahdaily.com .com 76
19,342 bold.dk .dk 58
19,411 clien.net .net 79
19,415 laist.com .com 79
19,429 thenewstack.io .io 69
19,430 larepubliquedespyrenees.fr .fr 81
19,439 econsultancy.com .com 88
19,461 shamela.ws .ws 55
19,478 french-stream.one .one 70
19,538 xn--l3ca4b6dsc1b.com .com 77
19,541 tecconcursos.com.br .br 88
19,542 caixin.com .com 65
19,563 blu-ray.com .com 69
19,564 playbill.com .com 68
19,576 moviesmod.at .at 77
19,614 daaz.com .com 43
19,624 fukuishimbun.co.jp .jp 76
19,631 spectrumlocalnews.com .com 45
19,632 coolrom.com .com 61
19,636 euobserver.com .com 86
19,662 aagmaal.kiwi .kiwi 69
19,672 wallhaven.cc .cc 45
19,678 skymovieshd.ceo .ceo 53
19,690 equidia.fr .fr 65
19,696 givemesport.com .com 83
19,703 sprzedajemy.pl .pl 75
19,719 moviesda.blue .blue 56
19,738 echodnia.eu .eu 79
19,755 100crorescompany.com .com 40
19,762 desixxx.blog .blog 65
19,810 boutique1st.com .com 56
19,836 nezavisne.com .com 69
19,860 zawya.com .com 80
19,861 meteo.gr .gr 76
19,870 gazetawroclawska.pl .pl 79
19,882 geo.de .de 82
19,902 kuttymovies.red .red 52
19,906 artificialintelligenceact.eu .eu 62
19,918 mkvmoviespoint.uno .uno 72
19,957 9xflix.gdn .gdn 80
19,979 maxwinexch99.com .com 27
19,989 uyeshare.cc .cc 64
20,007 juraforum.de .de 52
20,030 badische-zeitung.de .de 56
20,033 tradera.com .com 75
20,037 flixtor.to .to 45
20,080 chroniclelive.co.uk .uk 47
20,152 au.int .int 63
20,158 bigdatacloud.net .net 97
20,166 skift.com .com 82
20,225 niemanlab.org .org 51
20,232 uv.mx .mx 75
20,281 3dporndude.com .com 60
20,282 chileautos.cl .cl 68
20,310 cna.com.tw .tw 89
20,318 supersport.hr .hr 51
20,332 simplywall.st .st 95
PreviousPage 6 of 15Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the CCBot reference, the full census, and the JSON endpoint.