AI crawler · registry

MistralAI-User

Mistral AI · Live retrieval · robots.txt token mistralai-user

honors robots.txtMeasured across 29,991 sites with a robots.txt policy record for this agent, out of 29,996 on record.
The measurement

2% of 29,991 measured domains disallow mistralai-user in robots.txt at the site root. Separately, and independently of robots.txt, 17.0% of 29,996 reachable domains refuse an AI crawler user agent at the network edge on two consecutive probes from different locations - a refusal the site's own robots.txt does not declare and its owner often has not chosen.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "MistralAI-User blocking rate", measured 2026-11-21. https://crawlcensus.com/bot/mistralai-user

2%
of measured sites block it
457
sites blocking
29,534
sites allowing
yes
honors robots.txt

What it is definition

Fetches pages on demand so Mistral's Vibe assistant can answer a question with live, cited web content.

MistralAI-User is operated by Mistral AI and classified here as live retrieval. These agents fetch a single URL because a user asked about it. Disallowing one means the assistant answers from an older copy or from someone else's summary instead of the live page. Several are documented as not bound by robots.txt, because the request is user-initiated rather than a crawl.

Disallowing mistralai-user stops the fetch that happens when a user asks Le Chat about one of your URLs. The answer still gets written, from an older copy or from a third party's description of the page, and you no longer control what it says.

robots.txt token
mistralai-user
Operator
Mistral AI
Purpose
Live retrieval
Honors robots.txt
yes
Documentation
https://docs.mistral.ai/robots/

Identity on the wire exact string

User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
Verify your own edge is not refusing it
curl -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)' https://example.com/

Replace the host with your own. A 403, a challenge page or an empty body means the edge is blocking the agent whatever robots.txt says.

How to allow it, how to block it robots.txt

Allow it
User-agent: mistralai-user
Allow: /

Absent any rule, the agent is already allowed, so this group only matters when your * group disallows something. Naming the agent replaces the * group for it entirely, so repeat here every disallow you still want to apply to it.

Block it
User-agent: mistralai-user
Disallow: /

Disallowing mistralai-user stops the fetch that happens when a user asks Le Chat about one of your URLs. The answer still gets written, from an older copy or from a third party's description of the page, and you no longer control what it says.

Sites that block it 60 shown

DomainAI access scoreRank
amazon.com 40 26
tiktok.com 49 54
msn.com 41 62
amazonvideo.com 55 121
cnn.com 50 190
amazon.co.uk 40 318
amazon.de 40 377
amazon.co.jp 40 494
wired.com 81 533
uol.com.br 82 571
telegraph.co.uk 52 580
amazon.fr 40 609
amazon.co.za 40 623
dailymail.co.uk 61 627
amazon.ca 40 635
elpais.com 49 647
amazon.in 40 659
cnet.com 79 662
lemonde.fr 81 671
dailymail.com 61 681
primevideo.com 71 682
amazon.it 40 706
amazon.es 40 725
bild.de 81 763
people.com 44 801
amazon.com.au 40 812
theverge.com 77 815
nbcnews.com 59 823
amazon.com.br 40 833
lefigaro.fr 71 834
investopedia.com 44 913
corriere.it 63 955
elmundo.es 84 983
theconversation.com 62 1,019
amazon.com.mx 40 1,032
spiegel.de 64 1,042
marca.com 85 1,056
as.com 84 1,061
bfmtv.com 81 1,073
franceinfo.fr 60 1,098
ad.nl 52 1,190
nu.nl 49 1,191
furaffinity.net 39 1,223
hln.be 52 1,251
actu.fr 81 1,286
dagbladet.no 74 1,376
usnews.com 83 1,394
iltalehti.fi 58 1,409
eenadu.net 67 1,456
darkero.com 67 1,639
twkan.com 43 1,641
newyorker.com 73 1,793
zdnet.com 76 1,831
wikihow.com 76 1,835
fastcompany.com 53 1,892
themoviedb.org 74 1,984
ign.com 62 1,993
amazon.eg 40 2,006
mashable.com 61 2,010
justice.gov 54 2,084

See every measured domain that blocks MistralAI-User

This list is served from a cache rebuilt just now. It is discarded the moment a rescan records a policy change for this agent, so a site that has just changed its robots.txt will drop off it within one scan rather than waiting for the cache to expire.

Compare Mistral AI

Mistral AI runs 3 agents in this registry, and they do different jobs. Blocking one says nothing about the others.

Common questions answered

Does blocking MistralAI-User remove me from Le Chat?

Partly. MistralAI-User fetches a page only when a user asks about it, so disallowing it does not delist you from Le Chat. It does mean the answer is composed without your live page, from a stale copy or a third-party summary.

How do I verify MistralAI-User requests are genuine?

Mistral AI publishes neither reverse DNS names nor address ranges for MistralAI-User, so a request carrying this user agent cannot be authenticated. The string is trivially spoofed and is routinely reused by unrelated scrapers. Treat it as a hint, rate-limit by source address, and do not grant it access you would not grant an anonymous client.

Check your own site free

See whether your robots.txt admits MistralAI-User today, and whether your edge agrees with it.