Blocklist · Semrush

Sites that block SemrushBot-OCOB

1,059 of the 29,991 domains we have measured, 4%, disallow semrushbot-ocob at the site root in robots.txt. SemrushBot-OCOB collects content that may be used to train a model.

Every domain here has withheld its content from Semrush's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

1,059 of 29,991 measured domains (4%) disallow semrushbot-ocob at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block SemrushBot-OCOB", measured 2026-11-21. https://crawlcensus.com/blocked/semrushbot-ocob

The list 1,059 domains

RankDomainTLDAI access score
40,512 thetelegraphandargus.co.uk .uk 78
40,521 clearwebstats.com .com 68
40,545 convertingcolors.com .com 68
40,638 main-echo.de .de 64
40,678 aristeguinoticias.com .com 78
40,713 dwds.de .de 64
40,741 theblast.com .com 92
40,760 byflou.com .com 92
40,790 huxiu.com .com 50
40,796 elpasotimes.com .com 73
40,874 grid.id .id 64
40,956 joybuy.co.uk .uk 81
41,023 xrite.com .com 76
41,026 lnwshop.com .com 86
41,073 egm.gov.tr .tr 59
41,089 autobazar.eu .eu 91
41,121 lancashiretelegraph.co.uk .uk 78
41,140 rixos.com .com 69
41,161 quikr.com .com 65
41,204 elpuntavui.cat .cat 36
41,232 chipsmall.com .com 82
41,331 seair.co.in .in 55
41,490 flomni.com .com 50
41,519 greenme.it .it 82
41,542 oefb.at .at 50
41,646 goldposter.com .com 72
41,647 cjonline.com .com 73
41,653 donaukurier.de .de 78
41,678 usna.edu .edu 71
41,682 expedia.it .it 62
41,707 tshaonline.org .org 74
41,719 lyricsondemand.com .com 70
41,736 eadt.co.uk .uk 79
41,747 cheaptickets.com .com 65
41,786 rugs.com .com 86
41,800 planetf1.com .com 85
41,807 reefcentral.com .com 37
41,812 technologyadvice.com .com 79
41,818 digitalcombatsimulator.com .com 62
41,826 defcon.org .org 66
41,925 databridgemarketresearch.com .com 79
41,958 theledger.com .com 74
41,971 gallimard.fr .fr 77
41,999 slub-dresden.de .de 71
42,010 aprs.fi .fi 41
42,063 doorkeeper.jp .jp 70
42,076 iex.nl .nl 48
42,113 glasgowtimes.co.uk .uk 79
42,126 latestmodapks.com .com 75
42,160 tcpalm.com .com 73
42,260 rcgroups.com .com 41
42,293 tfd.com .com 69
42,322 studysmarter.de .de 77
42,333 localharvest.org .org 69
42,470 knightfrank.com .com 73
42,517 similarama.com .com 76
42,578 stolaf.edu .edu 84
42,584 gulf-times.com .com 88
42,594 desertcart.no .no 74
42,614 citizen-times.com .com 74
42,623 oglaszamy24.pl .pl 47
42,652 cars.co.za .za 70
42,804 techmeme.com .com 65
42,863 textstudio.com .com 74
42,883 ctrader.com .com 77
42,902 privacypolicytemplate.net .net 68
42,921 afad.gov.tr .tr 60
42,925 pixta.jp .jp 78
42,928 bruker.com .com 66
42,935 ugmk-telecom.ru .ru 63
42,972 uzinfocom.uz .uz 72
42,981 viewbug.com .com 83
43,095 aclweb.org .org 66
43,138 jobserve.com .com 69
43,207 tv4play.se .se 77
43,260 exchange-rates.org .org 81
43,268 candlepowerforums.com .com 61
43,302 seacoastonline.com .com 69
43,452 katestube.com .com 73
43,471 forvismazars.com .com 72
43,480 remoteok.com .com 76
43,562 fazwaz.com .com 71
43,578 pixmania.com .com 88
43,633 rexel.fr .fr 69
43,668 behostings.net .net 73
43,673 ojogo.pt .pt 69
43,728 termsandconditionsgenerator.com .com 73
43,909 gipuzkoa.eus .eus 70
43,946 jeopardylabs.com .com 61
43,975 iau.org .org 63
43,983 teambelgium.net .net 59
43,986 nn.de .de 83
44,026 gainesville.com .com 66
44,086 aboutyou.ch .ch 58
44,163 codebuddy.cn .cn 60
44,183 barcablaugranes.com .com 68
44,228 notionpress.com .com 64
44,246 rubylane.com .com 29
44,252 braunschweiger-zeitung.de .de 65
44,255 cagesideseats.com .com 72
44,280 thenewsminute.com .com 85
44,332 jizzbunker.com .com 69
44,403 58pic.com .com 52
44,483 angusrobertson.com.au .au 84
44,513 esalah.com .com 81
44,606 vidaxl.de .de 88
44,772 pinchofyum.com .com 91
44,807 gofeminin.de .de 74
44,811 ecrater.com .com 66
44,874 news-leader.com .com 62
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,035 freitag.de .de 90
45,055 wildskirts.com .com 67
45,064 erepublik.com .com 62
45,073 cyclegear.com .com 93
45,159 lansingstatejournal.com .com 62
45,195 natgeotv.com .com 81
45,244 wikifeet.com .com 49
45,266 ysu.edu .edu 77
45,284 athletics.app .app 65
45,316 eumetsat.int .int 80
45,349 wp.de .de 65
45,367 ftvnews.com.tw .tw 53
45,423 gtplanet.net .net 82
45,440 nrz.de .de 80
45,452 landwirt.com .com 94
45,536 bvb-alyans.ru .ru 80
45,538 gold.de .de 79
45,578 chugoku-np.co.jp .jp 80
45,604 beaconjournal.com .com 66
45,615 confidentialphonelookup.com .com 60
45,641 jpcycles.com .com 90
45,660 avtoto.ru .ru 69
45,680 greenvilleonline.com .com 62
45,688 bigl.ua .ua 78
45,842 awm.gov.au .au 72
45,876 nogizaka46.com .com 66
45,888 livre-rare-book.com .com 72
45,901 okaz.com.sa .sa 83
45,905 kob.com .com 84
45,989 aftenbladet.no .no 78
46,033 waptrick.com .com 65
46,125 stuller.com .com 89
46,139 vidaxl.nl .nl 87
46,160 mayoral.com .com 58
46,175 frag-christel.de .de 57
46,209 rosebikes.de .de 81
46,210 i3investor.com .com 53
46,231 registerguard.com .com 66
46,256 megaknihy.cz .cz 59
46,290 benu.cz .cz 71
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,361 ceeol.com .com 66
46,490 djmag.com .com 65
46,621 f1000research.com .com 74
46,671 edurev.in .in 89
46,753 grfc.ru .ru 50
46,779 miarroba.com .com 68
46,793 quickanddirtytips.com .com 80
46,814 ripoffreport.com .com 88
46,886 privacy.org.nz .nz 68
46,923 alisweb.org .org 39
47,029 lostarmour.info .info 51
47,039 sluurpy.com .com 89
47,096 vidaxl.fr .fr 88
47,125 oswego.edu .edu 79
47,173 eurogamer.de .de 57
47,184 lvm.de .de 94
47,206 jojowiki.com .com 83
47,237 roomies.com .com 76
47,285 bengo4.com .com 86
47,391 erzrf.ru .ru 76
47,455 folhavitoria.com.br .br 78
47,526 allgaeuer-zeitung.de .de 81
47,541 kabanchik.ua .ua 74
47,616 njump.me .me 69
47,723 selar.com .com 54
47,741 salzburg24.at .at 85
47,755 pigu.lt .lt 88
47,796 topekalibraries.info .info 46
47,884 vcstar.com .com 62
47,938 pornicom.com .com 73
47,948 theafricareport.com .com 55
47,986 davidrumsey.com .com 72
48,008 skepticalscience.com .com 68
48,074 grubstreet.com .com 81
48,132 awesomebooks.com .com 68
48,169 flowcode.com .com 95
48,238 marktjagd.de .de 74
48,369 newsnow.com .com 79
48,380 wsls.com .com 85
48,387 coro.net .net 89
48,453 atlantichealth.org .org 75
48,513 airnav.com .com 53
48,550 argusleader.com .com 62
48,595 muragon.com .com 85
48,689 naiz.eus .eus 75
48,697 cottages.com .com 91
PreviousPage 5 of 6Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the SemrushBot-OCOB reference, the full census, and the JSON endpoint.