Blocklist · Anthropic

Sites that block Claude-SearchBot

463 of the 29,991 domains we have measured, 2%, disallow claude-searchbot at the site root in robots.txt. Claude-SearchBot builds the index that decides whether the operator's assistant cites a page.

Every domain here has removed itself from Anthropic's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

The measurement

463 of 29,991 measured domains (2%) disallow claude-searchbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Claude-SearchBot", measured 2026-11-21. https://crawlcensus.com/blocked/claude-searchbot

The list 463 domains

RankDomainTLDAI access score
15,269 gamesindustry.biz .biz 57
15,401 desmoinesregister.com .com 74
15,509 diariovasco.com .com 82
15,557 cincinnati.com .com 74
15,944 rtl.be .be 82
16,103 dweb.link .link 65
16,164 christianitytoday.com .com 89
16,182 sermonaudio.com .com 71
16,209 ipfs.io .io 65
16,254 unil.ch .ch 83
16,317 ilmattino.it .it 62
16,377 decider.com .com 79
16,428 e24.no .no 78
16,564 mapgenie.io .io 59
16,664 lightningmaps.org .org 51
16,814 whattoexpect.com .com 65
16,846 porofessor.gg .gg 55
17,042 northjersey.com .com 74
17,183 semafor.com .com 72
17,432 kyoto-np.co.jp .jp 82
17,455 twincities.com .com 81
17,469 palmbeachpost.com .com 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,619 nst.com.my .my 42
17,694 manager-magazin.de .de 63
17,886 agrarheute.com .com 75
18,280 hoerzu.de .de 45
18,309 mans.io .io 60
18,442 empireonline.com .com 62
18,595 trycloudflare.com .com 52
18,863 dailynews.com .com 76
18,883 courier-journal.com .com 74
19,312 pinayflix.uk .uk 46
19,342 bold.dk .dk 58
19,563 blu-ray.com .com 69
19,624 fukuishimbun.co.jp .jp 76
19,852 taximaxim.ru .ru 69
20,362 thedodo.com .com 70
20,364 derwesten.de .de 83
20,958 pressdemocrat.com .com 80
21,038 mainpost.de .de 80
21,139 ledevoir.com .com 70
21,267 seher.no .no 70
21,823 desy.de .de 62
22,007 vg247.com .com 58
22,028 distrowatch.com .com 44
22,365 petitesannonces.ch .ch 67
22,623 mcall.com .com 78
22,987 syakouba.com .com 56
23,084 app.com .com 69
23,157 mlnba.fun .fun 54
23,288 catbox.moe .moe 59
24,014 askmen.com .com 47
24,022 twinkl.co.uk .uk 37
24,102 knoxnews.com .com 74
24,141 cntraveller.com .com 81
24,190 jacksonville.com .com 74
24,199 sciam.com .com 78
24,233 glamourmagazine.co.uk .uk 81
24,525 zin.ru .ru 41
24,567 newspicks.com .com 44
24,593 vogue.in .in 64
24,733 humanite.fr .fr 87
24,882 uol.com .com 82
24,892 eldiariomontanes.es .es 82
24,946 vogue.fr .fr 81
24,961 miaminewtimes.com .com 73
24,977 wired.it .it 83
25,048 tokyo-np.co.jp .jp 81
25,115 pilotonline.com .com 79
25,133 myrecipes.com .com 77
25,184 kompasiana.com .com 85
25,637 celebritynetworth.com .com 86
25,839 delawareonline.com .com 74
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
26,114 democratandchronicle.com .com 73
26,444 pap.pl .pl 39
26,637 brigitte.de .de 79
26,853 tallahassee.com .com 74
26,935 fd.nl .nl 73
26,949 vocus.cc .cc 78
27,723 geek.com .com 38
27,838 mtvuutiset.fi .fi 72
28,170 suedkurier.de .de 81
28,616 amzn.com .com 40
28,678 thenational.scot .scot 81
28,757 federalnewsnetwork.com .com 86
28,775 vogue.de .de 65
28,794 eastbaytimes.com .com 79
28,950 acronymfinder.com .com 65
29,048 gq-magazin.de .de 66
29,139 u-cursos.cl .cl 51
29,390 kompas.id .id 64
29,402 mmafighting.com .com 68
29,549 freiepresse.de .de 86
29,553 hankookilbo.com .com 84
29,634 westword.com .com 77
29,821 providencejournal.com .com 73
29,884 techtudo.com.br .br 84
30,052 lasprovincias.es .es 82
30,237 clarionledger.com .com 74
30,387 bsc.es .es 37
30,703 wickedlocal.com .com 75
30,737 diariosur.es .es 82
30,773 rheinpfalz.de .de 77
30,810 kentonline.co.uk .uk 76
30,986 commercialappeal.com .com 74
31,020 flexmls.com .com 89
31,023 phoenixnewtimes.com .com 71
31,032 howlongtobeat.com .com 49
31,138 ideal.es .es 82
31,176 edp24.co.uk .uk 79
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,352 sportsbusinessjournal.com .com 53
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,786 thespec.com .com 64
31,871 diariodesevilla.es .es 84
32,267 theargus.co.uk .uk 79
32,388 rgj.com .com 73
32,389 sepe.es .es 73
32,437 hoopshype.com .com 70
32,579 dallasobserver.com .com 77
32,617 flo.com.tr .tr 52
32,790 tatler.com .com 82
32,803 oxfordmail.co.uk .uk 79
32,918 laverdad.es .es 82
33,793 observer.co.uk .uk 74
34,097 vogue.es .es 65
34,309 unionleader.com .com 63
34,445 monoskop.org .org 43
34,750 mailplus.co.uk .uk 44
34,847 sydsvenskan.se .se 52
34,961 volksfreund.de .de 83
34,973 naplesnews.com .com 73
35,058 gaoding.com .com 62
35,062 dailypress.com .com 79
35,543 news-journalonline.com .com 73
35,759 houseandgarden.co.uk .uk 81
36,120 elcomercio.es .es 82
36,334 ytmnd.com .com 41
36,436 liquor.com .com 77
36,539 nd-aktuell.de .de 78
36,575 dailyecho.co.uk .uk 81
36,778 cwmars.org .org 43
36,827 verdi.de .de 58
37,150 taximaxim.com .com 60
37,547 bt.no .no 77
37,700 rutracker.net .net 43
38,007 wired.jp .jp 78
38,129 11freunde.de .de 58
38,148 news-press.com .com 73
38,327 inforum.com .com 70
38,409 igt.com .com 48
38,507 bournemouthecho.co.uk .uk 79
38,668 stimme.de .de 72
38,814 mas.to .to 31
38,876 elnortedecastilla.es .es 82
39,349 hoy.es .es 82
39,706 indiehackers.com .com 54
39,806 mercateo.com .com 48
39,824 werstreamt.es .es 59
40,068 intelligentgolf.co.uk .uk 66
40,339 kompas.tv .tv 76
40,512 thetelegraphandargus.co.uk .uk 78
40,638 main-echo.de .de 64
40,796 elpasotimes.com .com 73
40,874 grid.id .id 64
40,876 cntraveller.in .in 81
40,968 kontan.co.id .id 77
41,088 ispe.org .org 67
41,098 gqmagazine.fr .fr 82
41,121 lancashiretelegraph.co.uk .uk 78
41,204 elpuntavui.cat .cat 36
41,647 cjonline.com .com 73
41,721 canarias7.es .es 83
41,736 eadt.co.uk .uk 79
41,760 dailybreeze.com .com 76
41,872 vanityfair.fr .fr 66
41,892 icook.tw .tw 89
41,947 vogue.co.jp .jp 65
41,958 theledger.com .com 74
41,962 okinawatimes.co.jp .jp 79
41,999 slub-dresden.de .de 71
42,076 iex.nl .nl 48
42,113 glasgowtimes.co.uk .uk 79
42,160 tcpalm.com .com 73
42,293 tfd.com .com 69
42,614 citizen-times.com .com 74
42,698 sol.no .no 74
42,728 wishtv.com .com 80
42,919 tcrf.net .net 29
43,251 sufficientvelocity.com .com 73
43,302 seacoastonline.com .com 69
43,452 katestube.com .com 73
43,587 admagazine.fr .fr 81
43,666 tasty.co .co 79
PreviousPage 2 of 3Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Claude-SearchBot reference, the full census, and the JSON endpoint.