Blocklist · Semrush

Sites that block SemrushBot-OCOB

1,059 of the 29,991 domains we have measured, 4%, disallow semrushbot-ocob at the site root in robots.txt. SemrushBot-OCOB collects content that may be used to train a model.

Every domain here has withheld its content from Semrush's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

1,059 of 29,991 measured domains (4%) disallow semrushbot-ocob at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block SemrushBot-OCOB", measured 2026-11-21. https://crawlcensus.com/blocked/semrushbot-ocob

The list 1,059 domains

RankDomainTLDAI access score
30,658 barstoolsports.com .com 73
30,672 thegameroom.org .org 91
30,680 claro.com.gt .gt 93
30,703 wickedlocal.com .com 75
30,708 transferwise.com .com 87
30,755 infranken.de .de 89
30,837 couchbase.com .com 86
30,902 fact-finder.de .de 77
30,953 infobus.eu .eu 87
30,986 commercialappeal.com .com 74
30,992 huffingtonpost.it .it 83
30,995 kicktipp.es .es 64
31,004 kicktipp.pl .pl 63
31,006 n-jav.com .com 84
31,020 flexmls.com .com 89
31,032 howlongtobeat.com .com 49
31,176 edp24.co.uk .uk 79
31,189 flightconnections.com .com 49
31,191 h-net.org .org 59
31,213 filmpornocompleto.com .com 78
31,249 kicktipp.at .at 63
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,352 sportsbusinessjournal.com .com 53
31,367 en-academic.com .com 72
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,601 prodelibre.com.ar .ar 84
31,786 thespec.com .com 64
31,879 bookmaker-ratings.ru .ru 90
31,911 bigmarker.com .com 76
32,018 everything2.com .com 68
32,208 millenniumhotels.com .com 52
32,267 theargus.co.uk .uk 79
32,354 esrb.org .org 96
32,365 revzilla.com .com 95
32,388 rgj.com .com 73
32,437 hoopshype.com .com 70
32,456 currencyrate.today .today 67
32,478 la7.it .it 89
32,561 skapiec.pl .pl 77
32,571 ccm.net .net 91
32,617 flo.com.tr .tr 52
32,629 aacap.org .org 69
32,642 dreevoo.com .com 66
32,713 opovo.com.br .br 85
32,718 athletic.net .net 68
32,784 useit.com .com 72
32,803 oxfordmail.co.uk .uk 79
32,806 tvnet.lv .lv 79
32,807 wien.info .info 79
33,054 biologicaldiversity.org .org 66
33,141 form.run .run 68
33,204 loyverse.com .com 84
33,268 thebetterindia.com .com 74
33,280 uni-magdeburg.de .de 88
33,340 ovgu.de .de 88
33,411 lematin.ch .ch 84
33,562 myporntube.com .com 86
33,568 icecat.it .it 44
33,738 ruv.is .is 59
33,799 bollywoodhungama.com .com 92
33,843 docin.com .com 72
33,909 pololu.com .com 66
33,970 spitogatos.gr .gr 39
34,013 nihaojewelry.com .com 72
34,023 followfollow.com .com 73
34,044 rwe.com .com 83
34,113 kentek.ru .ru 83
34,147 valueyourmusic.com .com 73
34,171 photo-ac.com .com 50
34,185 atpress.ne.jp .jp 91
34,231 head-fi.org .org 56
34,253 indexbox.io .io 84
34,306 canadapost.ca .ca 74
34,310 coolrom.com.au .au 62
34,387 breakingnews.ie .ie 71
34,403 libris.ro .ro 93
34,445 monoskop.org .org 43
34,483 privacypolicyonline.com .com 69
34,564 tigerdroppings.com .com 76
34,589 shabdkosh.com .com 83
34,703 seksitreffit.fi .fi 69
34,716 plainproxies.com .com 92
34,783 teamtalk.com .com 89
34,792 fccid.io .io 75
34,818 securitm.ru .ru 92
34,870 mel.org .org 55
34,973 naplesnews.com .com 73
34,974 topagrar.com .com 85
35,029 zbozi.cz .cz 83
35,058 gaoding.com .com 62
35,158 jyllands-posten.dk .dk 70
35,171 artbasel.com .com 83
35,209 joybuy.de .de 81
35,340 relines.ru .ru 84
35,364 road.cc .cc 84
35,404 mapadeapellidos.com .com 75
35,443 thisdaylive.com .com 72
35,480 ac-illust.com .com 83
35,519 youglish.com .com 59
35,543 news-journalonline.com .com 73
35,643 e15.cz .cz 86
35,763 penguin.de .de 88
35,803 housepricesintheuk.co.uk .uk 74
35,805 hentaistube.com .com 68
35,861 ktv.jp .jp 67
35,879 betteruptime.com .com 74
35,881 initpro.ru .ru 66
35,889 jambase.com .com 70
35,896 studylight.org .org 76
35,933 hrs.com .com 82
36,077 xoilaclink-worldcup2026.tv .tv 95
36,078 xoilacttbd-worldcup2026.tv .tv 90
36,172 tfw2005.com .com 73
36,205 fair.org .org 54
36,334 ytmnd.com .com 41
36,369 gymmasteronline.com .com 63
36,372 eurobilltracker.com .com 65
36,539 nd-aktuell.de .de 78
36,550 savills.com .com 82
36,575 dailyecho.co.uk .uk 81
36,647 miragenews.com .com 86
36,748 westseattleblog.com .com 52
36,778 cwmars.org .org 43
36,841 privacypolicygenerator.info .info 69
36,867 unmask.com .com 62
36,895 uokik.gov.pl .pl 85
36,907 obit.ru .ru 94
37,035 kumu.io .io 66
37,076 wiu.edu .edu 72
37,150 taximaxim.com .com 60
37,209 23andme.org .org 98
37,303 lessentiel.lu .lu 84
37,367 mittelbayerische.de .de 77
37,415 tapinto.net .net 35
37,425 jamestown.org .org 82
37,522 twinkl.com .com 37
37,547 bt.no .no 77
37,552 videochatmix.com .com 88
37,566 numberempire.com .com 73
37,853 teamsideline.com .com 70
37,969 fsfe.org .org 74
37,998 gvm.com.tw .tw 97
38,053 dresden.de .de 59
38,068 bully-manga.com .com 81
38,129 11freunde.de .de 58
38,148 news-press.com .com 73
38,213 expedia.co.jp .jp 64
38,327 inforum.com .com 70
38,406 icecat.us .us 61
38,409 igt.com .com 48
38,476 jarir.com .com 77
38,507 bournemouthecho.co.uk .uk 79
38,510 hktvmall.com .com 67
38,532 tek-tips.com .com 56
38,576 euroweeklynews.com .com 88
38,648 wfmu.org .org 54
38,677 nvi.gov.tr .tr 60
38,707 pinkvilla.com .com 75
38,711 casinotest2.com .com 75
38,712 simplebooking.it .it 74
38,715 nvent.com .com 91
38,739 thueringer-allgemeine.de .de 82
38,740 easyfundraising.org.uk .uk 79
38,804 muzce.com .com 76
38,813 gayporn.com .com 89
38,856 salidzini.lv .lv 66
38,960 x-star.com .com 82
38,973 designcrowd.com .com 76
39,206 futurezone.de .de 86
39,211 sakhaday.ru .ru 87
39,281 teltarif.de .de 86
39,393 zakzak.co.jp .jp 72
39,437 at-s.com .com 81
39,583 opusdei.org .org 77
39,586 mindfactory.de .de 37
39,629 liveauctionworld.com .com 63
39,712 vsbattles.com .com 68
39,761 shopware.com .com 92
39,806 mercateo.com .com 48
39,824 werstreamt.es .es 59
39,852 svenskakyrkan.se .se 77
39,950 dholding.ru .ru 56
39,991 101hotels.com .com 83
40,025 jackdaniels.com .com 74
40,065 saudigazette.com.sa .sa 86
40,068 intelligentgolf.co.uk .uk 66
40,095 walletinvestor.com .com 87
40,305 seia.org .org 70
40,317 iza.ne.jp .jp 79
40,329 antpedia.com .com 57
40,339 kompas.tv .tv 76
40,360 noen.at .at 79
40,363 pockettactics.com .com 89
40,392 vanguardia.com.mx .mx 84
40,401 porn.biz .biz 70
40,405 art2arts.co.uk .uk 79
40,487 opendatabot.ua .ua 58
40,497 thepeninsulaqatar.com .com 86
PreviousPage 4 of 6Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the SemrushBot-OCOB reference, the full census, and the JSON endpoint.