Blocklist · Common Crawl Foundation

Sites that block CCBot

2,881 of the 29,991 domains we have measured, 10%, disallow ccbot at the site root in robots.txt. CCBot archives content.

Every domain here has withheld its content from Common Crawl Foundation's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,881 of 29,991 measured domains (10%) disallow ccbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block CCBot", measured 2026-11-21. https://crawlcensus.com/blocked/ccbot

The list 2,881 domains

RankDomainTLDAI access score
3,198 metro.co.uk .uk 59
3,208 n-able.com .com 84
3,213 fool.com .com 96
3,224 tagesschau.de .de 80
3,253 billboard.com .com 78
3,276 myanimelist.net .net 70
3,286 flipboard.com .com 64
3,291 cosmopolitan.com .com 79
3,295 goodhousekeeping.com .com 80
3,305 geeksforgeeks.org .org 85
3,320 unep.org .org 71
3,342 lifehacker.com .com 65
3,347 consumerreports.org .org 72
3,368 nzherald.co.nz .nz 56
3,371 nymag.com .com 74
3,418 deadline.com .com 81
3,433 thefreedictionary.com .com 71
3,437 tagesspiegel.de .de 74
3,457 telstra.com .com 88
3,487 jio.com .com 89
3,499 amazon.se .se 40
3,509 dictionary.com .com 90
3,520 amazon.ae .ae 40
3,524 chron.com .com 54
3,533 vanityfair.com .com 81
3,542 orf.at .at 69
3,576 abc.es .es 81
3,587 br.de .de 86
3,595 politico.eu .eu 62
3,598 amazon.nl .nl 40
3,603 macomnet.ru .ru 63
3,608 case-battle.id .id 48
3,617 boston.com .com 87
3,633 ndr.de .de 79
3,642 yahoo.net .net 67
3,662 filmyzilla52.com .com 55
3,676 handelsblatt.com .com 82
3,686 moviesda34.com .com 62
3,700 moviezwap.land .land 52
3,711 rexify.com.ng .ng 59
3,712 filmyzilla46.com .com 55
3,717 ansa.it .it 85
3,730 oregonstate.edu .edu 63
3,771 yle.fi .fi 85
3,796 ardmediathek.de .de 73
3,800 ebsco.com .com 65
3,808 toyota.com .com 60
3,869 pcworld.com .com 84
3,909 aol.de .de 65
3,916 news24.com .com 56
3,918 rte.ie .ie 83
3,934 menshealth.com .com 75
3,952 gitea.com .com 69
3,963 nos.nl .nl 69
3,966 bostonglobe.com .com 83
4,061 screenrant.com .com 84
4,065 seekingalpha.com .com 91
4,070 ilsole24ore.com .com 78
4,079 vaticannews.va .va 44
4,083 stern.de .de 83
4,084 uiowa.edu .edu 70
4,108 howtogeek.com .com 84
4,161 kompas.com .com 77
4,181 sbs.com.au .au 83
4,190 zoominfo.com .com 90
4,205 emojipedia.org .org 68
4,304 tvtropes.org .org 55
4,326 wiki.gg .gg 67
4,333 derstandard.at .at 56
4,334 blender.org .org 63
4,354 computerworld.com .com 83
4,398 flipsnack.com .com 94
4,407 20minutos.es .es 87
4,431 swr.de .de 76
4,484 mdr.de .de 76
4,493 wdr.de .de 86
4,497 gazeta.pl .pl 74
4,512 amazon.com.tr .tr 40
4,532 biorxiv.org .org 44
4,551 yna.co.kr .kr 87
4,553 20minutes.fr .fr 71
4,579 srf.ch .ch 87
4,609 123rf.com .com 43
4,658 rg.ru .ru 68
4,684 msnbc.com .com 75
4,744 patch.com .com 78
4,749 teacherspayteachers.com .com 69
4,753 xda-developers.com .com 83
4,791 collider.com .com 83
4,816 polygon.com .com 83
4,866 channel4.com .com 56
4,897 komoot.com .com 72
4,925 cadenaser.com .com 78
4,929 onlamp.com .com 80
4,965 vinted.co.uk .uk 91
4,987 dr.dk .dk 81
5,037 imageshack.com .com 71
5,057 deccanherald.com .com 64
5,068 gq.com .com 81
5,096 nine.com.au .au 71
5,103 tik.porn .porn 66
5,146 everydayhealth.com .com 60
5,153 theage.com.au .au 75
5,158 mercurynews.com .com 78
5,205 lapresse.ca .ca 65
5,223 oilprice.com .com 90
5,229 eltiempo.com .com 85
5,251 dallasnews.com .com 54
5,268 ora.com .com 94
5,272 depositphotos.com .com 63
5,274 csmonitor.com .com 76
5,277 bookshop.org .org 74
5,293 turbify.com .com 85
5,297 chinatimes.com .com 64
5,310 kotaku.com .com 89
5,323 cbr.com .com 83
5,357 larazon.es .es 81
5,395 makeuseof.com .com 84
5,401 mediamarkt.de .de 90
5,413 expressen.se .se 61
5,455 gamerant.com .com 81
5,457 diigo.com .com 35
5,463 rainews.it .it 53
5,472 sudouest.fr .fr 89
5,488 wallapop.com .com 60
5,523 architecturaldigest.com .com 83
5,547 amazon.ie .ie 40
5,552 onfinality.io .io 97
5,609 rtl.fr .fr 87
5,678 dn.se .se 60
5,684 theepochtimes.com .com 80
5,709 streamable.com .com 38
5,714 francetvinfo.fr .fr 53
5,722 hola.com .com 90
5,726 chefkoch.de .de 81
5,748 ovh.us .us 89
5,749 autoscout24.de .de 86
5,752 sankei.com .com 76
5,769 degruyter.com .com 61
5,770 boingboing.net .net 79
5,775 tv2.no .no 70
5,786 cloudpath.live .live 44
5,788 ibtimes.com .com 84
5,839 codecanyon.net .net 84
5,846 runescape.wiki .wiki 81
5,874 hs.fi .fi 68
5,876 dmitriy-mevedev-tor.top .top 53
5,879 cio.com .com 86
5,895 movieweb.com .com 84
5,904 windstream.net .net 53
5,912 pagesix.com .com 80
5,936 qobuz.com .com 79
5,948 suntimes.com .com 81
5,993 eonline.com .com 81
6,006 insider.com .com 94
6,019 vinted.de .de 67
6,020 giga.de .de 81
6,032 rtl.de .de 85
6,035 opaque.net .net 39
6,042 sporcle.com .com 87
6,062 nhk.jp .jp 64
6,064 delfi.lt .lt 83
6,065 tutsplus.com .com 85
6,088 futura-sciences.com .com 93
6,089 azcentral.com .com 74
6,111 ledauphine.com .com 82
6,125 denverpost.com .com 78
6,159 instiz.net .net 58
6,180 tf1info.fr .fr 84
6,185 trustedshops.com .com 87
6,228 sbnation.com .com 69
6,245 raiplay.it .it 55
6,246 eurogamer.net .net 57
6,275 leprogres.fr .fr 82
6,288 vulture.com .com 72
6,301 iata.org .org 68
6,308 wwd.com .com 78
6,335 royalroad.com .com 48
6,340 vinted.it .it 68
6,342 vinted.pl .pl 86
6,352 tv2.dk .dk 74
6,353 bbcgoodfood.com .com 84
6,356 freep.com .com 73
6,361 ici.fr .fr 87
6,365 icons8.com .com 74
6,386 huffingtonpost.co.uk .uk 82
6,391 sex-studentki.live .live 56
6,421 chavesnamao.com.br .br 80
6,446 mumsnet.com .com 79
6,462 uni-frankfurt.de .de 90
6,473 sfchronicle.com .com 54
6,482 airbridge.io .io 96
6,504 telstra.com.au .au 86
6,512 telegram.com .com 73
6,513 sandiegouniontribune.com .com 78
6,522 eater.com .com 59
6,530 theintercept.com .com 86
6,535 literotica.com .com 85
6,539 astro.com .com 37
6,544 rtvslo.si .si 65
PreviousPage 2 of 15Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the CCBot reference, the full census, and the JSON endpoint.