Blocklist · Semrush

Sites that block SemrushBot-OCOB

1,059 of the 29,991 domains we have measured, 4%, disallow semrushbot-ocob at the site root in robots.txt. SemrushBot-OCOB collects content that may be used to train a model.

Every domain here has withheld its content from Semrush's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

1,059 of 29,991 measured domains (4%) disallow semrushbot-ocob at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block SemrushBot-OCOB", measured 2026-11-21. https://crawlcensus.com/blocked/semrushbot-ocob

The list 1,059 domains

RankDomainTLDAI access score
10,268 css-tricks.com .com 77
10,271 anonymised.io .io 97
10,279 klingai.com .com 78
10,291 usmagazine.com .com 87
10,379 megagroup.ru .ru 90
10,400 morgenpost.de .de 81
10,414 dailykos.com .com 86
10,470 babycenter.com .com 62
10,576 manuals.plus .plus 98
10,585 anyflip.com .com 56
10,673 teleparty.com .com 54
10,852 embrapa.br .br 63
10,874 sendpulse.com .com 95
10,885 livestrong.com .com 80
10,894 augsburger-allgemeine.de .de 81
10,911 lwn.net .net 69
10,913 wyborcza.pl .pl 66
10,934 huffingtonpost.fr .fr 81
10,987 qu.ax .ax 63
11,109 hamburg.de .de 61
11,127 fronius.com .com 75
11,200 aftenposten.no .no 76
11,217 businessinsider.de .de 76
11,230 tau.ac.il .il 71
11,349 posteo.de .de 67
11,394 forecast7.com .com 45
11,405 svd.se .se 78
11,457 wnp.pl .pl 65
11,504 gamblingtherapy.org .org 87
11,516 cults3d.com .com 91
11,659 abendblatt.de .de 65
11,762 unistra.fr .fr 89
12,021 zahid.host .host 96
12,023 r7.com .com 86
12,111 techpowerup.com .com 50
12,114 allafrica.com .com 89
12,214 opencollective.com .com 71
12,250 globalso.com .com 64
12,334 krebsonsecurity.com .com 68
12,344 fringster.com .com 61
12,495 thegatewaypundit.com .com 85
12,533 indystar.com .com 73
12,703 uaserials.my .my 79
12,736 liveleak.com .com 47
12,743 extremetech.com .com 64
12,753 tvguide.com .com 70
12,797 golem.de .de 65
12,799 curbed.com .com 73
12,856 famikura-store.jp .jp 49
12,883 ahindian.com .com 54
12,965 uni-bielefeld.de .de 77
12,970 rtp.pt .pt 76
12,972 noticiasaominuto.com .com 85
12,990 nybooks.com .com 80
13,018 uwm.edu .edu 91
13,115 musicbrainz.org .org 71
13,243 snu.ac.kr .kr 70
13,320 timesnownews.com .com 66
13,333 wtop.com .com 74
13,579 barcelona.cat .cat 82
13,660 thrillist.com .com 70
13,667 dispatch.com .com 74
13,750 academic.ru .ru 72
13,772 khan.co.kr .kr 72
13,825 heraldscotland.com .com 81
13,871 ilex.by .by 59
13,879 tennessean.com .com 74
13,898 zakon.kz .kz 88
14,008 wponetap.com .com 94
14,055 observador.pt .pt 83
14,066 pourvous.nl .nl 72
14,079 wccftech.com .com 80
14,090 simon.com .com 31
14,103 delgarm.com .com 89
14,107 influxdata.com .com 90
14,209 toomics.com .com 72
14,259 advertising.com .com 78
14,285 bobolike.com .com 68
14,329 aboutyou.com .com 60
14,361 tournamentsoftware.com .com 66
14,449 sil.org .org 64
14,512 listennotes.com .com 64
14,724 simplypsychology.org .org 80
14,727 alfajertv.com .com 73
14,733 waz.de .de 82
14,742 osti.gov .gov 68
14,805 redalyc.org .org 52
14,826 rockpapershotgun.com .com 58
14,861 winscp.net .net 71
14,945 asha.org .org 65
15,017 mhanational.org .org 88
15,084 bestprice.gr .gr 97
15,174 pcgamesn.com .com 83
15,194 elifesciences.org .org 79
15,233 thieme-connect.com .com 39
15,269 gamesindustry.biz .biz 57
15,352 aaas.org .org 65
15,389 weatherspark.com .com 40
15,391 netzwelt.de .de 84
15,401 desmoinesregister.com .com 74
15,418 err.ee .ee 86
15,476 computerbild.de .de 81
15,517 journaldunet.com .com 89
15,539 aboutyou.de .de 60
15,547 privacypolicies.com .com 74
15,557 cincinnati.com .com 74
15,559 jyu.fi .fi 77
15,570 finna.fi .fi 54
15,686 postimees.ee .ee 66
15,838 caddy.community .community 68
15,858 alexistogel.lol .lol 73
16,082 t3n.de .de 78
16,231 nationstates.net .net 43
16,254 unil.ch .ch 83
16,283 positivepsychology.com .com 76
16,290 duden.de .de 71
16,421 dejure.org .org 72
16,428 e24.no .no 78
16,556 tourvisor.ru .ru 91
16,564 mapgenie.io .io 59
16,664 lightningmaps.org .org 51
16,795 vozpopuli.com .com 87
16,814 whattoexpect.com .com 65
16,944 macpaw.com .com 73
17,030 canadapost-postescanada.ca .ca 74
17,042 northjersey.com .com 74
17,116 jn.pt .pt 66
17,155 thefa.com .com 76
17,183 semafor.com .com 72
17,306 life.ru .ru 81
17,404 poipiku.com .com 71
17,469 palmbeachpost.com .com 73
17,489 dukeupress.edu .edu 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,561 mothercare.co.id .id 55
17,624 fourmilab.ch .ch 54
17,694 manager-magazin.de .de 63
17,819 oasis-open.org .org 79
17,886 agrarheute.com .com 75
17,989 clickondetroit.com .com 86
18,198 489ban.net .net 90
18,214 xxxbunker.com .com 81
18,280 hoerzu.de .de 45
18,286 gasbuddy.com .com 67
18,303 blackview.hk .hk 73
18,309 mans.io .io 60
18,362 signalhire.com .com 87
18,447 italki.com .com 93
18,489 elcolombiano.com .com 83
18,490 click2houston.com .com 85
18,552 getyarn.io .io 49
18,626 irishracing.com .com 82
18,662 mailenable.com .com 73
18,825 jrj.com.cn .cn 70
18,870 scotusblog.com .com 83
18,874 altibbi.com .com 94
18,883 courier-journal.com .com 74
18,894 courrierinternational.com .com 83
18,939 qgis.org .org 74
18,992 tokmanni.fi .fi 85
19,014 lovart.ai .ai 51
19,131 computerbase.de .de 89
19,182 clickorlando.com .com 85
19,193 wacoal.jp .jp 67
19,199 nttdata.com .com 91
19,272 insurancejournal.com .com 91
19,287 superbet.pl .pl 50
19,342 bold.dk .dk 58
19,411 clien.net .net 79
19,487 mondaq.com .com 64
19,553 filmix.gg .gg 75
19,563 blu-ray.com .com 69
19,632 coolrom.com .com 61
19,656 vacationstogo.com .com 69
19,679 eheya.net .net 86
19,703 sprzedajemy.pl .pl 75
19,823 sulekha.com .com 77
19,852 taximaxim.ru .ru 69
20,018 qsl.net .net 63
20,051 sinonim.org .org 59
20,120 openpr.com .com 66
20,214 teenxy.com .com 72
20,261 sportsport.ba .ba 85
20,279 halaxy.com .com 91
20,299 bankingnews.gr .gr 70
20,337 onelook.com .com 51
20,354 information.com .com 87
20,362 thedodo.com .com 70
20,364 derwesten.de .de 83
20,368 caratlane.com .com 79
20,383 youzan.com .com 89
20,460 mt.co.kr .kr 93
20,467 23andme.com .com 98
20,590 japanesebeauties.one .one 48
20,610 roliascan.com .com 89
20,693 lento.pl .pl 85
20,761 nlc.hu .hu 85
20,762 informatic.ru .ru 61
20,845 wnd.com .com 90
PreviousPage 2 of 6Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the SemrushBot-OCOB reference, the full census, and the JSON endpoint.