Blocklist · Cohere

Sites that block cohere-training-data-crawler

615 of the 29,991 domains we have measured, 2%, disallow cohere-training-data-crawler at the site root in robots.txt. cohere-training-data-crawler collects content that may be used to train a model.

Every domain here has withheld its content from Cohere's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

615 of 29,991 measured domains (2%) disallow cohere-training-data-crawler at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block cohere-training-data-crawler", measured 2026-11-21. https://crawlcensus.com/blocked/cohere-training-data-crawler

The list 615 domains

RankDomainTLDAI access score
49,501 sj-r.com .com 64
49,574 montgomeryadvertiser.com .com 62
49,802 pazar3.mk .mk 73
49,895 butenunbinnen.de .de 81
50,185 miricanvas.com .com 89
50,278 burlingtonfreepress.com .com 62
50,312 mujerhoy.com .com 80
50,338 ohtuleht.ee .ee 65
50,367 blipfoto.com .com 57
50,394 ardaudiothek.de .de 85
50,520 womansworld.com .com 85
50,684 gq.com.mx .mx 72
50,916 motorcycle.com .com 83
50,933 nwaonline.com .com 73
50,982 nrtool.st .st 54
PreviousPage 4 of 4

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the cohere-training-data-crawler reference, the full census, and the JSON endpoint.