AI crawler · registry

Claude-SearchBot

Anthropic · Answer index · robots.txt token claude-searchbot

honors robots.txtMeasured across 29,991 sites with a robots.txt policy record for this agent, out of 29,996 on record.
The measurement

2% of 29,991 measured domains disallow claude-searchbot in robots.txt at the site root. Separately, and independently of robots.txt, 17.0% of 29,996 reachable domains refuse an AI crawler user agent at the network edge on two consecutive probes from different locations - a refusal the site's own robots.txt does not declare and its owner often has not chosen.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Claude-SearchBot blocking rate", measured 2026-11-21. https://crawlcensus.com/bot/claude-searchbot

2%
of measured sites block it
463
sites blocking
29,528
sites allowing
yes
honors robots.txt

What it is definition

Indexes content to improve the relevance and accuracy of Claude's search results.

Claude-SearchBot is operated by Anthropic and classified here as answer index. These agents build the index an answer engine queries at question time. Disallowing one removes the site from those answers and from the citations that link back to it. Training is governed by separate tokens, so this is the expensive block, not the protective one.

Disallowing claude-searchbot removes the site from the index Claude search queries when it answers a question. The engine can then neither quote nor link the page, which is a traffic decision rather than a licensing one. Model training is governed by separate tokens.

robots.txt token
claude-searchbot
Operator
Anthropic
Purpose
Answer index
Honors robots.txt
yes
Documentation
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler

Identity on the wire exact string

User-agent string
Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)
Verify your own edge is not refusing it
curl -A 'Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)' https://example.com/

Replace the host with your own. A 403, a challenge page or an empty body means the edge is blocking the agent whatever robots.txt says.

How to allow it, how to block it robots.txt

Allow it
User-agent: claude-searchbot
Allow: /

Absent any rule, the agent is already allowed, so this group only matters when your * group disallows something. Naming the agent replaces the * group for it entirely, so repeat here every disallow you still want to apply to it.

Block it
User-agent: claude-searchbot
Disallow: /

Disallowing claude-searchbot removes the site from the index Claude search queries when it answers a question. The engine can then neither quote nor link the page, which is a traffic decision rather than a licensing one. Model training is governed by separate tokens.

Sites that block it 60 shown

DomainAI access scoreRank
amazon.com 40 26
tiktok.com 49 54
msn.com 41 62
nytimes.com 48 157
cnn.com 50 190
theguardian.com 60 193
bbc.com 76 207
bbc.co.uk 77 226
launchpad.net 52 260
amazon.co.uk 40 318
amazon.de 40 377
unsplash.com 64 397
cnbc.com 54 457
quora.com 46 479
amazon.co.jp 40 494
wired.com 81 533
uol.com.br 82 571
telegraph.co.uk 52 580
amazon.fr 40 609
amazon.co.za 40 623
dailymail.co.uk 61 627
amazon.ca 40 635
elpais.com 49 647
amazon.in 40 659
cnet.com 79 662
lemonde.fr 81 671
dailymail.com 61 681
amazon.it 40 706
amazon.es 40 725
apnews.com 49 787
people.com 44 801
amazon.com.au 40 812
theverge.com 77 815
nbcnews.com 59 823
amazon.com.br 40 833
lefigaro.fr 71 834
investopedia.com 44 913
corriere.it 63 955
elmundo.es 84 983
theconversation.com 62 1,019
huffpost.com 85 1,027
amazon.com.mx 40 1,032
spiegel.de 64 1,042
marca.com 85 1,056
as.com 84 1,061
bfmtv.com 81 1,073
nypost.com 81 1,083
franceinfo.fr 60 1,098
economist.com 48 1,157
aftonbladet.se 77 1,189
ad.nl 52 1,190
nu.nl 49 1,191
vg.no 79 1,208
furaffinity.net 39 1,223
hln.be 52 1,251
huffingtonpost.com 85 1,275
actu.fr 81 1,286
f95zone.to 59 1,300
dagbladet.no 74 1,376
iltalehti.fi 58 1,409

See every measured domain that blocks Claude-SearchBot

This list is served from a cache rebuilt just now. It is discarded the moment a rescan records a policy change for this agent, so a site that has just changed its robots.txt will drop off it within one scan rather than waiting for the cache to expire.

Compare Anthropic

Anthropic runs 4 agents in this registry, and they do different jobs. Blocking one says nothing about the others.

Common questions answered

Does blocking Claude-SearchBot remove me from Claude search?

Yes, in effect. Claude-SearchBot builds the index Claude search reads at question time, so disallowing it takes the site out of those answers and out of the citations that link back to it. This is the block that costs traffic.

How do I verify Claude-SearchBot requests are genuine?

Anthropic publishes the address ranges its crawlers fetch from, linked from its crawler support article. Compare the source address of the request against that list. There is no reverse DNS name to check.

Check your own site free

See whether your robots.txt admits Claude-SearchBot today, and whether your edge agrees with it.