Blocklist · Cohere

Sites that block cohere-training-data-crawler

615 of the 29,991 domains we have measured, 2%, disallow cohere-training-data-crawler at the site root in robots.txt. cohere-training-data-crawler collects content that may be used to train a model.

Every domain here has withheld its content from Cohere's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

615 of 29,991 measured domains (2%) disallow cohere-training-data-crawler at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block cohere-training-data-crawler", measured 2026-11-21. https://crawlcensus.com/blocked/cohere-training-data-crawler

The list 615 domains

RankDomainTLDAI access score
29,402 mmafighting.com .com 68
29,692 piliapp.com .com 65
29,821 providencejournal.com .com 73
30,004 sfstandard.com .com 72
30,052 lasprovincias.es .es 82
30,237 clarionledger.com .com 74
30,387 bsc.es .es 37
30,443 dobrzemieszkaj.pl .pl 61
30,703 wickedlocal.com .com 75
30,737 diariosur.es .es 82
30,974 techadvisor.com .com 85
30,986 commercialappeal.com .com 74
31,032 howlongtobeat.com .com 49
31,138 ideal.es .es 82
31,149 idealhome.co.uk .uk 82
31,176 edp24.co.uk .uk 79
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,638 marieclaire.co.uk .uk 84
31,782 mastodon.online .online 49
31,786 thespec.com .com 64
31,850 avsforum.com .com 43
31,871 diariodesevilla.es .es 84
32,032 prx.org .org 74
32,267 theargus.co.uk .uk 79
32,388 rgj.com .com 73
32,437 hoopshype.com .com 70
32,498 toptenreviews.com .com 87
32,706 marianne.net .net 67
32,713 opovo.com.br .br 85
32,718 athletic.net .net 68
32,735 rsi.ch .ch 63
32,790 tatler.com .com 82
32,803 oxfordmail.co.uk .uk 79
32,806 tvnet.lv .lv 79
32,918 laverdad.es .es 82
33,494 theanalyst.com .com 88
33,568 icecat.it .it 44
33,644 tv4.se .se 76
33,738 ruv.is .is 59
33,934 carlsen.de .de 71
34,012 womanandhome.com .com 85
34,097 vogue.es .es 65
34,309 unionleader.com .com 63
34,310 coolrom.com.au .au 62
34,445 monoskop.org .org 43
34,750 mailplus.co.uk .uk 44
34,847 sydsvenskan.se .se 52
34,973 naplesnews.com .com 73
34,986 ruhrnachrichten.de .de 75
35,058 gaoding.com .com 62
35,062 dailypress.com .com 79
35,326 swr3.de .de 74
35,364 road.cc .cc 84
35,543 news-journalonline.com .com 73
35,759 houseandgarden.co.uk .uk 81
35,763 penguin.de .de 88
35,953 redflagdeals.com .com 44
35,955 oekotest.de .de 86
36,120 elcomercio.es .es 82
36,334 ytmnd.com .com 41
36,436 liquor.com .com 77
36,539 nd-aktuell.de .de 78
36,575 dailyecho.co.uk .uk 81
36,646 cyclingweekly.com .com 85
36,803 tennis.com .com 59
37,297 storyblocks.com .com 70
37,522 twinkl.com .com 37
37,547 bt.no .no 77
37,821 golfmonthly.com .com 83
38,007 wired.jp .jp 78
38,129 11freunde.de .de 58
38,148 news-press.com .com 73
38,193 ardsounds.de .de 85
38,327 inforum.com .com 70
38,406 icecat.us .us 61
38,409 igt.com .com 48
38,507 bournemouthecho.co.uk .uk 79
38,519 kochbar.de .de 62
38,668 stimme.de .de 72
38,739 thueringer-allgemeine.de .de 82
38,876 elnortedecastilla.es .es 82
39,206 futurezone.de .de 86
39,349 hoy.es .es 82
39,393 zakzak.co.jp .jp 72
39,471 vwvortex.com .com 43
39,657 korben.info .info 77
39,706 indiehackers.com .com 54
39,824 werstreamt.es .es 59
40,317 iza.ne.jp .jp 79
40,328 bntu.by .by 80
40,339 kompas.tv .tv 76
40,512 thetelegraphandargus.co.uk .uk 78
40,638 main-echo.de .de 64
40,713 dwds.de .de 64
40,796 elpasotimes.com .com 73
40,868 sayweee.com .com 72
40,874 grid.id .id 64
40,876 cntraveller.in .in 81
40,904 nesn.com .com 73
41,098 gqmagazine.fr .fr 82
41,121 lancashiretelegraph.co.uk .uk 78
41,290 theweek.co.uk .uk 84
41,336 archinect.com .com 68
41,349 planet-wissen.de .de 71
41,519 greenme.it .it 82
41,542 oefb.at .at 50
41,590 at5.nl .nl 70
41,603 lci.fr .fr 84
41,614 thebulwark.com .com 54
41,647 cjonline.com .com 73
41,736 eadt.co.uk .uk 79
41,760 dailybreeze.com .com 76
41,826 defcon.org .org 66
41,872 vanityfair.fr .fr 66
41,947 vogue.co.jp .jp 65
41,958 theledger.com .com 74
41,962 okinawatimes.co.jp .jp 79
42,054 lecker.de .de 88
42,076 iex.nl .nl 48
42,113 glasgowtimes.co.uk .uk 79
42,160 tcpalm.com .com 73
42,260 rcgroups.com .com 41
42,262 advrider.com .com 41
42,582 decanter.com .com 85
42,614 citizen-times.com .com 74
42,651 computerwoche.de .de 85
42,698 sol.no .no 74
43,207 tv4play.se .se 77
43,268 candlepowerforums.com .com 61
43,302 seacoastonline.com .com 69
43,310 3djuegos.com .com 93
43,548 tvmovie.de .de 89
43,587 admagazine.fr .fr 81
43,605 zonerama.com .com 30
43,974 ybw.com .com 87
44,026 gainesville.com .com 66
44,183 barcablaugranes.com .com 68
44,252 braunschweiger-zeitung.de .de 65
44,255 cagesideseats.com .com 72
44,349 nhnieuws.nl .nl 74
44,807 gofeminin.de .de 74
44,874 news-leader.com .com 62
44,878 rtvnoord.nl .nl 73
44,908 mora.jp .jp 45
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,158 autoguide.com .com 87
45,159 lansingstatejournal.com .com 62
45,240 vogue.mx .mx 65
45,272 rtvutrecht.nl .nl 73
45,316 eumetsat.int .int 80
45,349 wp.de .de 65
45,440 nrz.de .de 80
45,578 chugoku-np.co.jp .jp 80
45,604 beaconjournal.com .com 66
45,680 greenvilleonline.com .com 62
45,762 autozeitung.de .de 86
45,771 larioja.com .com 83
45,846 amazon.jp .jp 40
45,905 kob.com .com 84
45,918 revistagq.com .com 81
45,989 aftenbladet.no .no 78
46,076 bitgo.com .com 94
46,231 registerguard.com .com 66
46,314 backpacker.com .com 78
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,539 estar.jp .jp 62
46,805 omroepwest.nl .nl 73
46,923 alisweb.org .org 39
46,949 schulferien.org .org 51
46,990 fixya.com .com 43
47,029 lostarmour.info .info 51
47,173 eurogamer.de .de 57
47,299 rtvdrenthe.nl .nl 73
47,526 allgaeuer-zeitung.de .de 81
47,624 lacucinaitaliana.it .it 81
47,659 nationalgeographic.de .de 82
47,796 topekalibraries.info .info 46
47,850 espinof.com .com 94
47,884 vcstar.com .com 62
47,909 moneyweek.com .com 86
47,930 diariodecadiz.es .es 67
48,074 grubstreet.com .com 81
48,444 vogue.com.tw .tw 81
48,480 xatakamovil.com .com 92
48,550 argusleader.com .com 62
48,777 mtv.fi .fi 76
48,858 otz.de .de 65
49,067 gqitalia.it .it 64
49,194 mailonsunday.co.uk .uk 61
49,271 eltern.de .de 72
49,279 presstelegram.com .com 80
49,343 hd.se .se 51
49,420 gds.it .it 77
49,447 businessinsider.es .es 87
49,464 ucc.edu.gh .gh 94
PreviousPage 3 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the cohere-training-data-crawler reference, the full census, and the JSON endpoint.