Blocklist · Anthropic

Sites that block Claude-User

498 of the 29,991 domains we have measured, 2%, disallow claude-user at the site root in robots.txt. Claude-User fetches a page live when a user asks the assistant for it.

Every domain here has removed itself from Anthropic's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

The measurement

498 of 29,991 measured domains (2%) disallow claude-user at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Claude-User", measured 2026-11-21. https://crawlcensus.com/blocked/claude-user

The list 498 domains

RankDomainTLDAI access score
39,824 werstreamt.es .es 59
40,068 intelligentgolf.co.uk .uk 66
40,184 estantevirtual.com.br .br 91
40,339 kompas.tv .tv 76
40,512 thetelegraphandargus.co.uk .uk 78
40,638 main-echo.de .de 64
40,796 elpasotimes.com .com 73
40,830 xink.io .io 78
40,874 grid.id .id 64
40,876 cntraveller.in .in 81
40,968 kontan.co.id .id 77
41,088 ispe.org .org 67
41,098 gqmagazine.fr .fr 82
41,121 lancashiretelegraph.co.uk .uk 78
41,204 elpuntavui.cat .cat 36
41,603 lci.fr .fr 84
41,647 cjonline.com .com 73
41,736 eadt.co.uk .uk 79
41,760 dailybreeze.com .com 76
41,872 vanityfair.fr .fr 66
41,947 vogue.co.jp .jp 65
41,958 theledger.com .com 74
41,962 okinawatimes.co.jp .jp 79
42,076 iex.nl .nl 48
42,113 glasgowtimes.co.uk .uk 79
42,160 tcpalm.com .com 73
42,293 tfd.com .com 69
42,614 citizen-times.com .com 74
42,666 nhregister.com .com 54
42,698 sol.no .no 74
42,728 wishtv.com .com 80
42,919 tcrf.net .net 29
43,251 sufficientvelocity.com .com 73
43,302 seacoastonline.com .com 69
43,587 admagazine.fr .fr 81
43,627 tvrain.ru .ru 84
43,666 tasty.co .co 79
44,026 gainesville.com .com 66
44,183 barcablaugranes.com .com 68
44,255 cagesideseats.com .com 72
44,280 thenewsminute.com .com 85
44,807 gofeminin.de .de 74
44,874 news-leader.com .com 62
44,878 rtvnoord.nl .nl 73
44,928 wgt.com .com 80
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,159 lansingstatejournal.com .com 62
45,240 vogue.mx .mx 65
45,272 rtvutrecht.nl .nl 73
45,316 eumetsat.int .int 80
45,578 chugoku-np.co.jp .jp 80
45,604 beaconjournal.com .com 66
45,680 greenvilleonline.com .com 62
45,771 larioja.com .com 83
45,846 amazon.jp .jp 40
45,918 revistagq.com .com 81
46,168 bizapedia.com .com 60
46,231 registerguard.com .com 66
46,301 redcafe.net .net 64
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,635 wikihow.life .life 76
46,805 omroepwest.nl .nl 73
46,923 alisweb.org .org 39
47,173 eurogamer.de .de 57
47,299 rtvdrenthe.nl .nl 73
47,526 allgaeuer-zeitung.de .de 81
47,624 lacucinaitaliana.it .it 81
47,796 topekalibraries.info .info 46
47,884 vcstar.com .com 62
48,074 grubstreet.com .com 81
48,404 lesoleil.com .com 69
48,444 vogue.com.tw .tw 81
48,550 argusleader.com .com 62
48,613 theoutdoorstrader.com .com 52
48,859 castro.fm .fm 67
49,059 citypopulation.de .de 54
49,065 ingramcontent.com .com 69
49,067 gqitalia.it .it 64
49,126 scientific.net .net 52
49,194 mailonsunday.co.uk .uk 61
49,271 eltern.de .de 72
49,279 presstelegram.com .com 80
49,343 hd.se .se 51
49,420 gds.it .it 77
49,501 sj-r.com .com 64
49,574 montgomeryadvertiser.com .com 62
49,695 pricesmart.com .com 61
49,925 aamulehti.fi .fi 66
50,278 burlingtonfreepress.com .com 62
50,312 mujerhoy.com .com 80
50,367 blipfoto.com .com 57
50,684 gq.com.mx .mx 72
50,933 nwaonline.com .com 73
51,009 treasure-maps.com .com 51
51,018 everydaycalculation.com .com 52
- articulo.mercadolibre.cl .cl 67
PreviousPage 3 of 3

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Claude-User reference, the full census, and the JSON endpoint.