Blocklist · Semrush

Sites that block SemrushBot-OCOB

1,059 of the 29,991 domains we have measured, 4%, disallow semrushbot-ocob at the site root in robots.txt. SemrushBot-OCOB collects content that may be used to train a model.

Every domain here has withheld its content from Semrush's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

1,059 of 29,991 measured domains (4%) disallow semrushbot-ocob at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block SemrushBot-OCOB", measured 2026-11-21. https://crawlcensus.com/blocked/semrushbot-ocob

The list 1,059 domains

RankDomainTLDAI access score
20,932 1kuji.com .com 69
20,967 699pic.com .com 52
20,989 emploi-territorial.fr .fr 66
21,019 sravni.ru .ru 83
21,024 hostlove.com .com 61
21,030 rugbyrama.fr .fr 89
21,038 mainpost.de .de 80
21,079 wort-suchen.de .de 81
21,081 netvasco.com.br .br 67
21,139 ledevoir.com .com 70
21,162 psytests.org .org 69
21,220 mindat.org .org 47
21,231 epochtimes.com .com 85
21,430 gamefa.com .com 95
21,505 lifehacker.ru .ru 85
21,517 jaya9bdt.com .com 70
21,544 telesurtv.net .net 89
21,549 callescort.org .org 70
21,564 plotek.pl .pl 75
21,600 pathosting.com .com 74
21,652 toernooi.nl .nl 69
21,683 popt.in .in 85
21,692 wikidex.net .net 74
21,694 rci.fm .fm 62
21,707 khronos.org .org 68
21,731 reklama5.mk .mk 44
21,742 capfun.com .com 89
21,778 cielo.com.br .br 91
21,823 desy.de .de 62
21,864 xvideosincesto.com .com 80
21,889 scoop.co.nz .nz 67
21,890 telerama.fr .fr 83
21,930 endnote.com .com 71
21,968 playtube.tv .tv 70
22,007 vg247.com .com 58
22,009 realescort.eu .eu 54
22,028 distrowatch.com .com 44
22,031 news4jax.com .com 85
22,035 dcu.ie .ie 75
22,132 pointi.jp .jp 82
22,148 smartico.ai .ai 97
22,155 savetik.co .co 69
22,193 vignanam.org .org 61
22,272 geograph.org.uk .uk 84
22,365 petitesannonces.ch .ch 67
22,459 leofame.com .com 67
22,479 ksat.com .com 85
22,525 tmcnet.com .com 49
22,584 blocklist.de .de 69
22,659 centraldepasajes.com.ar .ar 89
22,746 pazarium.com.tr .tr 88
22,767 itemfix.com .com 72
22,789 3seq.com .com 57
22,987 syakouba.com .com 56
23,044 adzuna.co.uk .uk 51
23,084 app.com .com 69
23,217 xnung.com .com 97
23,317 amaterky.sk .sk 75
23,334 xvideos.blog.br .br 91
23,417 testmozusercontent.com .com 68
23,613 nordbayern.de .de 81
23,660 asfinag.at .at 74
23,945 bitcoinmagazine.com .com 85
23,957 essential.gg .gg 73
23,972 kyobobook.co.kr .kr 84
23,983 nmu.edu .edu 79
24,000 icecat.biz .biz 44
24,014 askmen.com .com 47
24,022 twinkl.co.uk .uk 37
24,102 knoxnews.com .com 74
24,190 jacksonville.com .com 74
24,445 baamboozle.com .com 42
24,479 demandsage.com .com 90
24,525 zin.ru .ru 41
24,547 hhu.de .de 75
24,672 onetwotrip.com .com 62
24,680 canoe.ca .ca 90
24,710 tricolor.ru .ru 65
24,787 waikato.ac.nz .nz 89
24,882 uol.com .com 82
24,934 svtplay.se .se 78
24,983 sn.at .at 70
25,028 uni-greifswald.de .de 85
25,048 tokyo-np.co.jp .jp 81
25,078 freemalaysiatoday.com .com 92
25,126 nidirect.gov.uk .uk 77
25,150 servertown.ch .ch 80
25,227 soft112.com .com 74
25,386 uni-duesseldorf.de .de 75
25,453 research.com .com 70
25,476 podcastaddict.com .com 77
25,531 forebears.io .io 66
25,590 raceroster.com .com 89
25,638 truenetwork.ru .ru 70
25,839 delawareonline.com .com 74
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
25,939 moneydj.com .com 64
25,941 inmyroom.ru .ru 48
26,039 tadviser.ru .ru 73
26,097 km.ru .ru 72
26,114 democratandchronicle.com .com 73
26,210 kstp.com .com 83
26,244 superbet.rs .rs 50
26,411 lanit.ru .ru 69
26,426 polkachu.com .com 69
26,436 golfgenius.com .com 84
26,474 mbta.com .com 64
26,637 brigitte.de .de 79
26,653 bas.bg .bg 82
26,728 thueringen.de .de 77
26,843 additudemag.com .com 93
26,853 tallahassee.com .com 74
26,901 cmake.org .org 73
26,921 jpc.de .de 91
26,963 premiumoutlets.com .com 31
26,969 geekbuying.com .com 78
27,060 honeybook.com .com 93
27,082 lesswrong.com .com 66
27,126 domserver.de .de 81
27,140 torrentdownload.info .info 66
27,158 beds24.com .com 90
27,182 ultimahora.es .es 83
27,233 flipbuilder.com .com 75
27,256 amny.com .com 83
27,326 epark.jp .jp 56
27,354 munhwa.com .com 72
27,381 tradekey.com .com 82
27,422 affinity.co .co 95
27,517 bookofra-slot.fr .fr 85
27,525 bigporn.com .com 72
27,690 desertcart.com .com 58
27,723 geek.com .com 38
27,737 texasattorneygeneral.gov .gov 65
27,783 openwall.com .com 67
27,838 mtvuutiset.fi .fi 72
27,850 smoothcomp.com .com 50
27,941 wwf.de .de 78
27,999 earth.com .com 80
28,018 diia.gov.ua .ua 76
28,021 sr.ht .ht 55
28,048 udc.es .es 69
28,067 juming.com .com 32
28,083 varsitytutors.com .com 84
28,140 bdnews24.com .com 92
28,149 dotmed.com .com 72
28,170 suedkurier.de .de 81
28,220 icecat.se .se 52
28,271 booklog.jp .jp 69
28,290 icecat.dk .dk 56
28,324 eurekamag.com .com 88
28,328 ipshu.com .com 82
28,356 aecom.com .com 88
28,365 hoseasons.co.uk .uk 91
28,483 icollector.com .com 65
28,626 openfoodfacts.org .org 59
28,669 suzuri.jp .jp 93
28,678 thenational.scot .scot 81
28,792 uesp.net .net 55
28,849 unitel.bo .bo 75
28,930 championcross.jp .jp 62
28,945 kicktipp.com .com 64
28,950 acronymfinder.com .com 65
29,011 fraserinstitute.org .org 51
29,121 pucsp.br .br 55
29,139 u-cursos.cl .cl 51
29,173 tudosaladeaula.com .com 91
29,269 elmir.ua .ua 50
29,293 ceu.edu .edu 58
29,318 cookieconsent.com .com 85
29,390 kompas.id .id 64
29,402 mmafighting.com .com 68
29,430 xtgem.com .com 64
29,467 ascon.ru .ru 70
29,499 conjur.com.br .br 73
29,510 uvigo.es .es 56
29,553 hankookilbo.com .com 84
29,692 piliapp.com .com 65
29,701 thieme-connect.de .de 79
29,732 potaroo.net .net 65
29,821 providencejournal.com .com 73
29,887 insanejournal.com .com 51
30,004 sfstandard.com .com 72
30,032 lesanciennes.com .com 77
30,074 yugipedia.com .com 58
30,102 mutualart.com .com 91
30,196 youngchampion.jp .jp 62
30,237 clarionledger.com .com 74
30,273 centerblog.net .net 64
30,356 edaily.co.kr .kr 74
30,387 bsc.es .es 37
30,393 ecode360.com .com 44
30,431 rossmann.pl .pl 95
30,443 dobrzemieszkaj.pl .pl 61
30,480 modthesims.info .info 55
30,500 routard.com .com 78
30,516 rendez-vous.ru .ru 69
30,546 ala.org.au .au 75
30,573 govtschemes.in .in 75
30,609 heinonline.org .org 46
PreviousPage 3 of 6Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the SemrushBot-OCOB reference, the full census, and the JSON endpoint.