Blocklist · Cohere

Sites that block cohere-training-data-crawler

615 of the 29,991 domains we have measured, 2%, disallow cohere-training-data-crawler at the site root in robots.txt. cohere-training-data-crawler collects content that may be used to train a model.

Every domain here has withheld its content from Cohere's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

615 of 29,991 measured domains (2%) disallow cohere-training-data-crawler at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block cohere-training-data-crawler", measured 2026-11-21. https://crawlcensus.com/blocked/cohere-training-data-crawler

The list 615 domains

RankDomainTLDAI access score
10,885 livestrong.com .com 80
10,894 augsburger-allgemeine.de .de 81
10,933 rts.ch .ch 69
10,934 huffingtonpost.fr .fr 81
11,154 internshala.com .com 87
11,200 aftenposten.no .no 76
11,217 businessinsider.de .de 76
11,242 teenvogue.com .com 82
11,349 posteo.de .de 67
11,405 svd.se .se 78
11,457 wnp.pl .pl 65
11,516 cults3d.com .com 91
11,610 allure.com .com 84
11,659 abendblatt.de .de 65
11,714 hackmd.io .io 88
12,111 techpowerup.com .com 50
12,113 bostonherald.com .com 78
12,294 ally.com .com 94
12,377 delfi.ee .ee 57
12,533 indystar.com .com 73
12,542 epicurious.com .com 64
12,550 kiplinger.com .com 85
12,743 extremetech.com .com 64
12,799 curbed.com .com 73
12,924 syfy.com .com 80
12,972 noticiasaominuto.com .com 85
12,990 nybooks.com .com 80
13,091 elle.fr .fr 69
13,131 ranker.com .com 78
13,169 csoonline.com .com 86
13,201 hessenschau.de .de 87
13,237 clicrbs.com.br .br 80
13,355 marieclaire.com .com 85
13,398 gq-magazine.co.uk .uk 81
13,451 skyscrapercity.com .com 43
13,579 barcelona.cat .cat 82
13,660 thrillist.com .com 70
13,667 dispatch.com .com 74
13,783 creativebloq.com .com 87
13,825 heraldscotland.com .com 81
13,860 francebleu.fr .fr 87
13,871 ilex.by .by 59
13,879 tennessean.com .com 74
13,882 vermont.gov .gov 73
14,053 androidcentral.com .com 83
14,079 wccftech.com .com 80
14,090 simon.com .com 31
14,168 ncaa.com .com 85
14,374 delfi.lv .lv 80
14,709 dazeddigital.com .com 78
14,733 waz.de .de 82
14,809 auto-motor-und-sport.de .de 81
14,826 rockpapershotgun.com .com 58
14,872 mz.de .de 70
15,174 pcgamesn.com .com 83
15,218 statesman.com .com 51
15,269 gamesindustry.biz .biz 57
15,311 mcmaster.com .com 69
15,321 coursehero.com .com 76
15,372 letudiant.fr .fr 81
15,401 desmoinesregister.com .com 74
15,458 utopia.de .de 85
15,476 computerbild.de .de 81
15,509 diariovasco.com .com 82
15,557 cincinnati.com .com 74
15,686 postimees.ee .ee 66
16,082 t3n.de .de 78
16,086 nintendo.co.jp .jp 50
16,118 healio.com .com 69
16,134 homesandgardens.com .com 84
16,252 vivino.com .com 82
16,254 unil.ch .ch 83
16,290 duden.de .de 71
16,323 ard.de .de 76
16,428 e24.no .no 78
16,444 hometalk.com .com 63
16,564 mapgenie.io .io 59
16,664 lightningmaps.org .org 51
16,783 taipeitimes.com .com 66
16,795 vozpopuli.com .com 87
16,814 whattoexpect.com .com 65
17,042 northjersey.com .com 74
17,183 semafor.com .com 72
17,432 kyoto-np.co.jp .jp 82
17,455 twincities.com .com 81
17,469 palmbeachpost.com .com 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,694 manager-magazin.de .de 63
17,886 agrarheute.com .com 75
17,890 directoalpaladar.com .com 92
17,936 cinemablend.com .com 87
17,983 brightlocal.com .com 90
18,147 volksstimme.de .de 68
18,280 hoerzu.de .de 45
18,308 preyproject.com .com 93
18,442 empireonline.com .com 62
18,489 elcolombiano.com .com 83
18,522 thenews.com.pk .pk 65
18,608 sensacine.com .com 81
18,863 dailynews.com .com 76
18,883 courier-journal.com .com 74
18,894 courrierinternational.com .com 83
19,624 fukuishimbun.co.jp .jp 76
19,632 coolrom.com .com 61
19,690 equidia.fr .fr 65
19,860 zawya.com .com 80
19,891 musicradar.com .com 87
20,318 supersport.hr .hr 51
20,362 thedodo.com .com 70
20,364 derwesten.de .de 83
20,556 bravotv.com .com 74
20,580 daryo.uz .uz 76
20,718 itpro.com .com 83
20,761 nlc.hu .hu 85
20,847 gardeningknowhow.com .com 83
20,958 pressdemocrat.com .com 80
21,004 femina.hu .hu 82
21,038 mainpost.de .de 80
21,231 epochtimes.com .com 85
21,267 seher.no .no 70
21,481 futureplc.com .com 86
21,566 whowhatwear.com .com 82
21,809 cyclingnews.com .com 85
21,823 desy.de .de 62
21,828 wallpaper.com .com 85
21,889 scoop.co.nz .nz 67
22,007 vg247.com .com 58
22,028 distrowatch.com .com 44
22,216 airliners.net .net 44
22,623 mcall.com .com 78
23,084 app.com .com 69
23,154 rbb24.de .de 80
23,846 dailypost.ng .ng 83
23,875 imore.com .com 81
23,964 pcwelt.de .de 78
24,000 icecat.biz .biz 44
24,014 askmen.com .com 47
24,022 twinkl.co.uk .uk 37
24,102 knoxnews.com .com 74
24,141 cntraveller.com .com 81
24,157 whathifi.com .com 87
24,190 jacksonville.com .com 74
24,233 glamourmagazine.co.uk .uk 81
24,525 zin.ru .ru 41
24,567 newspicks.com .com 44
24,593 vogue.in .in 64
24,714 occrp.org .org 62
24,882 uol.com .com 82
24,892 eldiariomontanes.es .es 82
24,934 svtplay.se .se 78
24,946 vogue.fr .fr 81
24,977 wired.it .it 83
25,007 digitalcameraworld.com .com 86
25,031 franceculture.fr .fr 81
25,048 tokyo-np.co.jp .jp 81
25,115 pilotonline.com .com 79
25,133 myrecipes.com .com 77
25,478 franceinter.fr .fr 80
25,776 zinio.com .com 76
25,796 shopltk.com .com 83
25,800 trendencias.com .com 94
25,839 delawareonline.com .com 74
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
26,114 democratandchronicle.com .com 73
26,210 kstp.com .com 83
26,240 yogajournal.com .com 77
26,249 guitarworld.com .com 84
26,474 mbta.com .com 64
26,637 brigitte.de .de 79
26,806 golf.com .com 88
26,853 tallahassee.com .com 74
26,935 fd.nl .nl 73
26,963 premiumoutlets.com .com 31
27,078 mdzol.com .com 61
27,136 purepeople.com.br .br 85
27,182 ultimahora.es .es 83
27,256 amny.com .com 83
27,315 fourfourtwo.com .com 84
27,401 derstandard.de .de 56
27,696 staradvertiser.com .com 84
27,723 geek.com .com 38
27,838 mtvuutiset.fi .fi 72
27,999 earth.com .com 80
28,170 suedkurier.de .de 81
28,220 icecat.se .se 52
28,290 icecat.dk .dk 56
28,320 laptopmag.com .com 87
28,573 wpr.org .org 88
28,616 amzn.com .com 40
28,678 thenational.scot .scot 81
28,775 vogue.de .de 65
28,792 uesp.net .net 55
28,794 eastbaytimes.com .com 79
29,048 gq-magazin.de .de 66
29,115 gp.se .se 78
29,139 u-cursos.cl .cl 51
29,323 goop.com .com 66
29,390 kompas.id .id 64
PreviousPage 2 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the cohere-training-data-crawler reference, the full census, and the JSON endpoint.