Blocklist · Semrush

Sites that block SemrushBot-OCOB

1,059 of the 29,991 domains we have measured, 4%, disallow semrushbot-ocob at the site root in robots.txt. SemrushBot-OCOB collects content that may be used to train a model.

Every domain here has withheld its content from Semrush's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

1,059 of 29,991 measured domains (4%) disallow semrushbot-ocob at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block SemrushBot-OCOB", measured 2026-11-21. https://crawlcensus.com/blocked/semrushbot-ocob

The list 1,059 domains

RankDomainTLDAI access score
190 cnn.com .com 50
246 who.int .int 90
260 launchpad.net .net 52
342 issuu.com .com 66
399 aliexpress.com .com 81
473 arxiv.org .org 68
556 alrosa.ru .ru 68
570 eset.com .com 86
571 uol.com.br .br 82
662 cnet.com .com 79
671 lemonde.fr .fr 81
710 bandcamp.com .com 53
763 bild.de .de 81
815 theverge.com .com 77
834 lefigaro.fr .fr 71
942 repubblica.it .it 85
1,019 theconversation.com .com 62
1,030 wise.com .com 87
1,042 spiegel.de .de 64
1,054 n-tv.de .de 88
1,072 alphonso.tv .tv 76
1,073 bfmtv.com .com 81
1,147 psychologytoday.com .com 73
1,189 aftonbladet.se .se 77
1,208 vg.no .no 79
1,223 furaffinity.net .net 39
1,273 ladepeche.fr .fr 86
1,300 f95zone.to .to 59
1,446 ekstrabladet.dk .dk 67
1,471 bookmark.xxx .xxx 63
1,595 redwap.sex .sex 49
1,686 animanch.com .com 58
1,789 1c.ru .ru 79
1,790 notion.com .com 94
1,831 zdnet.com .com 76
1,984 themoviedb.org .org 74
1,993 ign.com .com 62
2,007 notion.so .so 93
2,010 mashable.com .com 61
2,230 meb.gov.tr .tr 66
2,353 kicktipp.de .de 63
2,401 jaya9bangladesh.com .com 70
2,423 pcmag.com .com 63
2,820 cookieyes.com .com 95
2,880 sueddeutsche.de .de 71
2,939 narod.ru .ru 72
2,971 prokerala.com .com 81
2,975 imageshack.us .us 71
2,990 about.me .me 72
3,285 thingiverse.com .com 40
3,342 lifehacker.com .com 65
3,371 nymag.com .com 74
3,433 thefreedictionary.com .com 71
3,440 hrw.org .org 82
3,463 postman.com .com 92
3,595 politico.eu .eu 62
3,603 macomnet.ru .ru 63
3,680 termsfeed.com .com 75
3,690 salon.com .com 90
3,916 news24.com .com 56
4,044 alltrails.com .com 85
4,120 wurl.com .com 89
4,161 kompas.com .com 77
4,271 goldapple.ru .ru 53
4,304 tvtropes.org .org 55
4,407 20minutos.es .es 87
4,497 gazeta.pl .pl 74
4,532 biorxiv.org .org 44
4,592 lastampa.it .it 81
4,666 preply.com .com 85
4,744 patch.com .com 78
4,852 vedomosti.ru .ru 89
4,864 kinoafisha.info .info 91
4,874 kommersant.ru .ru 91
4,927 iol.co.za .za 69
4,961 20min.ch .ch 84
5,015 join.chat .chat 92
5,037 imageshack.com .com 71
5,112 aap.org .org 76
5,146 everydayhealth.com .com 60
5,205 lapresse.ca .ca 65
5,357 larazon.es .es 81
5,490 superbet.ro .ro 54
5,678 dn.se .se 60
5,684 theepochtimes.com .com 80
5,685 linternaute.com .com 86
5,709 streamable.com .com 38
5,715 wolfram.com .com 92
5,752 sankei.com .com 76
5,846 runescape.wiki .wiki 81
5,901 lsu.edu .edu 76
6,032 rtl.de .de 85
6,089 azcentral.com .com 74
6,109 fsf.org .org 84
6,228 sbnation.com .com 69
6,246 eurogamer.net .net 57
6,288 vulture.com .com 72
6,352 tv2.dk .dk 74
6,356 freep.com .com 73
6,369 hessen.de .de 81
6,512 telegram.com .com 73
6,522 eater.com .com 59
6,539 astro.com .com 37
6,549 midilibre.fr .fr 86
6,682 rit.edu .edu 89
6,718 journaldesfemmes.fr .fr 84
6,727 nngroup.com .com 72
6,768 akakce.com .com 80
6,785 russianfood.com .com 68
6,793 svt.se .se 66
6,808 chunichi.co.jp .jp 71
6,816 maxroll.gg .gg 59
6,833 bergfex.at .at 81
6,865 illinois.gov .gov 67
6,886 pngtree.com .com 34
6,895 togetter.com .com 73
6,934 blesk.cz .cz 62
6,948 nicematin.com .com 77
7,039 newsnow.co.uk .uk 82
7,056 freeprivacypolicy.com .com 71
7,106 nrc.nl .nl 84
7,118 nme.com .com 81
7,127 hani.co.kr .kr 67
7,142 klix.ba .ba 84
7,161 vhx.tv .tv 72
7,178 sanspo.com .com 77
7,201 game8.jp .jp 72
7,225 ygosu.com .com 64
7,239 gala.de .de 76
7,345 rpp.pe .pe 90
7,355 bandlab.com .com 59
7,372 15min.lt .lt 74
7,397 fuckmoral.com .com 64
7,432 nouvelobs.com .com 87
7,556 lindependant.fr .fr 86
7,608 bundesregierung.de .de 87
7,671 epson.com .com 63
7,821 gala.fr .fr 90
7,856 popsugar.com .com 75
7,863 refinery29.com .com 78
7,933 tuya.com .com 89
7,941 sport.pl .pl 75
7,947 cnnamador.com .com 74
8,011 seesaa.net .net 48
8,032 metacritic.com .com 66
8,054 bebee.com .com 87
8,094 nefisyemektarifleri.com .com 76
8,142 bazos.cz .cz 78
8,144 myheritage.com .com 68
8,160 javhdporn.net .net 71
8,268 litmarket.ru .ru 87
8,287 javdock.com .com 65
8,326 xxxshame.com .com 65
8,380 veikkaus.fi .fi 67
8,435 thecut.com .com 84
8,474 69shuba.com .com 45
8,502 a1.net .net 77
8,543 jjgirls.com .com 50
8,555 onlinejobs.ph .ph 66
8,565 whois.com .com 87
8,601 bazos.sk .sk 78
8,681 cnn.gr .gr 87
8,721 ddnavi.com .com 72
8,747 st-andrews.ac.uk .uk 79
8,755 pudelek.pl .pl 89
8,763 mijnwoordenboek.nl .nl 73
8,800 sefan.ru .ru 43
8,803 libramemoria.com .com 91
8,815 olimpbet.kz .kz 44
8,830 portalsamorzadowy.pl .pl 69
8,883 manhuagui.com .com 61
8,889 snapsave.app .app 92
8,897 rynekzdrowia.pl .pl 64
8,911 danword.com .com 52
8,928 nottingham.ac.uk .uk 64
8,962 eboblack.com .com 89
9,037 sokmil.com .com 46
9,060 alo.bg .bg 71
9,062 bluewin.ch .ch 66
9,069 wbur.org .org 81
9,107 milfat.com .com 84
9,149 forth.gr .gr 60
9,299 masstamilan.dev .dev 70
9,339 detroitnews.com .com 68
9,360 priceoye.pk .pk 82
9,388 law.com .com 68
9,473 crex.com .com 80
9,487 riyasewana.com .com 78
9,574 classificadosx.net .net 91
9,619 modaselvim.com .com 62
9,659 arm.com .com 78
9,665 yakkun.com .com 68
9,718 diva-portal.org .org 49
9,754 qustodio.com .com 89
9,804 porno18.site .site 76
9,903 mk.co.kr .kr 82
9,925 iconify.design .design 55
9,948 xosothantai.mobi .mobi 75
10,042 fictionexpress.com .com 76
10,074 cope.es .es 86
Page 1 of 6Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the SemrushBot-OCOB reference, the full census, and the JSON endpoint.