Blocklist · Common Crawl Foundation

Sites that block CCBot

2,881 of the 29,991 domains we have measured, 10%, disallow ccbot at the site root in robots.txt. CCBot archives content.

Every domain here has withheld its content from Common Crawl Foundation's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

2,881 of 29,991 measured domains (10%) disallow ccbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block CCBot", measured 2026-11-22. https://crawlcensus.com/blocked/ccbot

The list 2,881 domains

RankDomainTLDAI access score
50,063 textileexchange.org .org 70
50,070 retrogames.cz .cz 60
50,076 dublinlive.ie .ie 47
50,141 umh.es .es 68
50,168 sabq.org .org 72
50,172 palworld-lab.com .com 65
50,173 andscape.com .com 86
50,180 banglanews24.com .com 72
50,185 miricanvas.com .com 89
50,210 rummyapklist.com .com 78
50,233 thefenomeno.com .com 74
50,240 moviezmad.cool .cool 71
50,278 burlingtonfreepress.com .com 62
50,312 mujerhoy.com .com 80
50,314 thesportzfyapps.com .com 83
50,338 ohtuleht.ee .ee 65
50,342 thenewslens.com .com 71
50,367 blipfoto.com .com 57
50,394 ardaudiothek.de .de 85
50,417 vidoyyo.space .space 26
50,418 gagadget.com .com 72
50,426 bolshoyvopros.ru .ru 70
50,452 natureetdecouvertes.com .com 84
50,461 gamioo5.com .com 56
50,478 scryfall.io .io 68
50,492 financialfilings.com .com 94
50,520 womansworld.com .com 85
50,548 fakenamegenerator.com .com 71
50,579 cleopatraslot.org .org 69
50,598 dailycal.org .org 74
50,682 commonsense.org .org 86
50,684 gq.com.mx .mx 72
50,718 frdic.com .com 49
50,732 vesty.co.il .il 91
50,738 grantland.com .com 57
50,761 adzuna.fr .fr 51
50,788 liverpool.com .com 47
50,803 partycity.com .com 87
50,844 dobreprogramy.pl .pl 92
50,854 rtcg.me .me 61
50,861 ascension.gg .gg 82
50,869 xn--4-wxfc3gwbi.com .com 77
50,877 movieinsider.com .com 70
50,891 damconuong.store .store 82
50,905 notice-facile.com .com 80
50,916 motorcycle.com .com 83
50,919 japantravel.com .com 79
50,933 nwaonline.com .com 73
50,946 dutasexx.hair .hair 66
50,953 alaskantf.com .com 45
50,960 hot.com .com 67
50,979 lopinion.fr .fr 81
50,982 nrtool.st .st 54
51,003 kenyans.co.ke .ke 78
51,009 treasure-maps.com .com 51
51,018 everydaycalculation.com .com 52
- 0daymusic.org .org 52
- aiproductads.app .app 95
- aitextcleaner.pro .pro 96
- asteroidsathome.net .net 64
- digibouquetai.com .com 95
- flashjournal.space .space 41
- headlinebeacon.site .site 43
- iptv-org.pro .pro 90
- iptvplaylist.app .app 88
- kemp-stokes-3.blogbright.net .net 46
- kling4ai.app .app 93
- kling5.art .art 93
- lodiguideph.com .com 75
- minimaxh4.app .app 92
- minimaxh5.com .com 92
- mp3tomidi.art .art 57
- musicvideoai.pro .pro 73
- myreadinglists.com .com 49
- samplenightlive.com .com 75
- sdgyoungleaders.org .org 84
- seedaud.io .io 77
- tarotai.chat .chat 96
- tds.net .net 53
- thecip.org .org 39
- veo4-video.art .art 77
- wan4ai.app .app 92
PreviousPage 15 of 15

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the CCBot reference, the full census, and the JSON endpoint.