Blocklist · Semrush

Sites that block SemrushBot-OCOB

1,059 of the 29,991 domains we have measured, 4%, disallow semrushbot-ocob at the site root in robots.txt. SemrushBot-OCOB collects content that may be used to train a model.

Every domain here has withheld its content from Semrush's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

1,059 of 29,991 measured domains (4%) disallow semrushbot-ocob at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block SemrushBot-OCOB", measured 2026-11-21. https://crawlcensus.com/blocked/semrushbot-ocob

The list 1,059 domains

RankDomainTLDAI access score
48,733 horiba.com .com 73
48,742 schoolandcollegelistings.com .com 56
48,775 homelight.com .com 89
48,777 mtv.fi .fi 76
48,843 auto-data.net .net 94
48,858 otz.de .de 65
48,859 castro.fm .fm 67
48,870 thecalculatorsite.com .com 69
48,890 emergencyemail.org .org 71
48,907 billboard-japan.com .com 69
48,954 ebth.com .com 78
48,982 aboutyou.hu .hu 60
48,999 joybuy.nl .nl 81
49,061 posfie.com .com 73
49,228 rtl.hu .hu 86
49,253 communityimpact.com .com 86
49,271 eltern.de .de 72
49,351 apolloduck.com .com 57
49,358 generator.email .email 89
49,393 disp.cc .cc 65
49,420 gds.it .it 77
49,434 damoang.net .net 80
49,447 businessinsider.es .es 87
49,501 sj-r.com .com 64
49,513 businessinsurance.com .com 69
49,574 montgomeryadvertiser.com .com 62
49,611 gamedesire.com .com 85
49,802 pazar3.mk .mk 73
49,886 everyjoe.com .com 73
49,924 express-shina.ru .ru 84
50,003 europeanfilmgateway.eu .eu 65
50,033 stadt-bremerhaven.de .de 76
50,050 timeform.com .com 78
50,120 estatesales.org .org 84
50,133 sud.ua .ua 72
50,215 newscast.jp .jp 87
50,278 burlingtonfreepress.com .com 62
50,358 nearcut.com .com 73
50,367 blipfoto.com .com 57
50,379 vaia.com .com 66
50,383 tedox.de .de 74
50,435 geoedge.com .com 90
50,471 rheumatology.org .org 76
50,487 tamilchristiansongs.in .in 67
50,514 claro.com.do .do 93
50,520 womansworld.com .com 85
50,544 selfhtml.org .org 71
50,727 namur.be .be 67
50,761 adzuna.fr .fr 51
50,791 escapadarural.com .com 90
50,826 yesform.com .com 84
50,867 goc.gov.tr .tr 59
50,879 promediateknologi.id .id 71
50,919 japantravel.com .com 79
50,933 nwaonline.com .com 73
50,982 nrtool.st .st 54
- asteroidsathome.net .net 64
- hhso.org .org 85
- tildes.net .net 69
PreviousPage 6 of 6

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the SemrushBot-OCOB reference, the full census, and the JSON endpoint.