AI crawler · registry

MistralAI-Index

Mistral AI · Answer index · robots.txt token mistralai-index

honors robots.txtMeasured across 29,991 sites with a robots.txt policy record for this agent, out of 29,996 on record.
The measurement

0.4% of 29,991 measured domains disallow mistralai-index in robots.txt at the site root. Separately, and independently of robots.txt, 17.0% of 29,996 reachable domains refuse an AI crawler user agent at the network edge on two consecutive probes from different locations - a refusal the site's own robots.txt does not declare and its owner often has not chosen.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "MistralAI-Index blocking rate", measured 2026-11-21. https://crawlcensus.com/bot/mistralai-index

0.4%
of measured sites block it
111
sites blocking
29,880
sites allowing
yes
honors robots.txt

What it is definition

Indexes content for Mistral search behind Vibe answers; not used for generative AI training.

MistralAI-Index is operated by Mistral AI and classified here as answer index. These agents build the index an answer engine queries at question time. Disallowing one removes the site from those answers and from the citations that link back to it. Training is governed by separate tokens, so this is the expensive block, not the protective one.

Disallowing mistralai-index removes the site from the index Mistral AI's AI products queries when it answers a question. The engine can then neither quote nor link the page, which is a traffic decision rather than a licensing one. Model training is governed by separate tokens.

robots.txt token
mistralai-index
Operator
Mistral AI
Purpose
Answer index
Honors robots.txt
yes
Documentation
https://docs.mistral.ai/robots/

Identity on the wire exact string

User-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)
Verify your own edge is not refusing it
curl -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)' https://example.com/

Replace the host with your own. A 403, a challenge page or an empty body means the edge is blocking the agent whatever robots.txt says.

How to allow it, how to block it robots.txt

Allow it
User-agent: mistralai-index
Allow: /

Absent any rule, the agent is already allowed, so this group only matters when your * group disallows something. Naming the agent replaces the * group for it entirely, so repeat here every disallow you still want to apply to it.

Block it
User-agent: mistralai-index
Disallow: /

Disallowing mistralai-index removes the site from the index Mistral AI's AI products queries when it answers a question. The engine can then neither quote nor link the page, which is a traffic decision rather than a licensing one. Model training is governed by separate tokens.

Sites that block it 60 shown

DomainAI access scoreRank
msn.com 41 62
lemonde.fr 81 671
people.com 44 801
investopedia.com 44 913
franceinfo.fr 60 1,098
actu.fr 81 1,286
wnacg.com 41 1,528
ign.com 62 1,993
techradar.com 87 2,215
livescience.com 85 2,585
allrecipes.com 44 3,184
flipboard.com 64 3,286
tomshardware.com 87 3,331
space.com 85 3,873
findagrave.com 41 3,906
kompas.com 77 4,161
verywellhealth.com 44 4,285
verywellmind.com 44 4,344
20minutes.fr 71 4,553
independent.ie 44 4,569
marthastewart.com 44 4,625
thespruce.com 44 4,644
eatingwell.com 44 4,747
telegraaf.nl 44 4,941
ew.com 44 4,983
pcgamer.com 85 5,026
everydayhealth.com 60 5,146
travelandleisure.com 44 5,171
diigo.com 35 5,457
dn.se 60 5,678
theepochtimes.com 80 5,684
seriouseats.com 44 5,686
thebalancemoney.com 44 5,708
francetvinfo.fr 53 5,714
rtl.nl 47 5,731
thoughtco.com 44 6,008
nieuwsblad.be 44 6,093
eurogamer.net 57 6,246
gamesradar.com 85 6,377
sandiegouniontribune.com 78 6,513
maxroll.gg 59 6,816
bhg.com 44 6,940
nicematin.com 77 6,948
realsimple.com 44 7,023
treehugger.com 44 7,276
lifewire.com 44 7,823
theweek.com 84 8,866
parents.com 44 9,485
findnovel.net 64 9,542
windowscentral.com 85 10,335
morgenpost.de 81 10,400
babycenter.com 62 10,470
thebalance.com 44 10,471
newspapers.com 41 10,678
livestrong.com 80 10,885
sme.sk 50 11,113
abendblatt.de 65 11,659
instyle.com 44 12,144
novelfire.net 64 12,441
kiplinger.com 85 12,550

See every measured domain that blocks MistralAI-Index

This list is served from a cache rebuilt just now. It is discarded the moment a rescan records a policy change for this agent, so a site that has just changed its robots.txt will drop off it within one scan rather than waiting for the cache to expire.

Compare Mistral AI

Mistral AI runs 3 agents in this registry, and they do different jobs. Blocking one says nothing about the others.

Common questions answered

Does blocking MistralAI-Index remove me from Mistral AI's AI products?

Yes, in effect. MistralAI-Index builds the index Mistral AI's AI products reads at question time, so disallowing it takes the site out of those answers and out of the citations that link back to it. This is the block that costs traffic.

How do I verify MistralAI-Index requests are genuine?

Mistral AI publishes neither reverse DNS names nor address ranges for MistralAI-Index, so a request carrying this user agent cannot be authenticated. The string is trivially spoofed and is routinely reused by unrelated scrapers. Treat it as a hint, rate-limit by source address, and do not grant it access you would not grant an anonymous client.

Check your own site free

See whether your robots.txt admits MistralAI-Index today, and whether your edge agrees with it.