AI crawler · registry

Meta-WebIndexer

Meta · Answer index · robots.txt token meta-webindexer

honors robots.txtMeasured across 29,991 sites with a robots.txt policy record for this agent, out of 29,996 on record.
The measurement

1% of 29,991 measured domains disallow meta-webindexer in robots.txt at the site root. Separately, and independently of robots.txt, 17.0% of 29,996 reachable domains refuse an AI crawler user agent at the network edge on two consecutive probes from different locations - a refusal the site's own robots.txt does not declare and its owner often has not chosen.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Meta-WebIndexer blocking rate", measured 2026-11-21. https://crawlcensus.com/bot/meta-webindexer

1%
of measured sites block it
402
sites blocking
29,589
sites allowing
yes
honors robots.txt

What it is definition

Indexes pages so Meta AI can cite and link them in its search answers.

Meta-WebIndexer is operated by Meta and classified here as answer index. These agents build the index an answer engine queries at question time. Disallowing one removes the site from those answers and from the citations that link back to it. Training is governed by separate tokens, so this is the expensive block, not the protective one.

Disallowing meta-webindexer removes the site from the index Meta's AI products queries when it answers a question. The engine can then neither quote nor link the page, which is a traffic decision rather than a licensing one. Model training is governed by separate tokens.

robots.txt token
meta-webindexer
Operator
Meta
Purpose
Answer index
Honors robots.txt
yes
Documentation
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/

Identity on the wire exact string

User-agent string
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
Verify your own edge is not refusing it
curl -A 'meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)' https://example.com/

Replace the host with your own. A 403, a challenge page or an empty body means the edge is blocking the agent whatever robots.txt says.

How to allow it, how to block it robots.txt

Allow it
User-agent: meta-webindexer
Allow: /

Absent any rule, the agent is already allowed, so this group only matters when your * group disallows something. Naming the agent replaces the * group for it entirely, so repeat here every disallow you still want to apply to it.

Block it
User-agent: meta-webindexer
Disallow: /

Disallowing meta-webindexer removes the site from the index Meta's AI products queries when it answers a question. The engine can then neither quote nor link the page, which is a traffic decision rather than a licensing one. Model training is governed by separate tokens.

Sites that block it 60 shown

DomainAI access scoreRank
amazon.com 40 26
amazonvideo.com 55 121
nytimes.com 48 157
amazon.co.uk 40 318
amazon.de 40 377
amazon.co.jp 40 494
ft.com 57 519
wired.com 81 533
amazon.fr 40 609
amazon.co.za 40 623
dailymail.co.uk 61 627
amazon.ca 40 635
amazon.in 40 659
cnet.com 79 662
lemonde.fr 81 671
dailymail.com 61 681
primevideo.com 71 682
amazon.it 40 706
amazon.es 40 725
people.com 44 801
amazon.com.au 40 812
nbcnews.com 59 823
amazon.com.br 40 833
investopedia.com 44 913
onet.pl 76 977
theconversation.com 62 1,019
amazon.com.mx 40 1,032
bfmtv.com 81 1,073
franceinfo.fr 60 1,098
nu.nl 49 1,191
furaffinity.net 39 1,223
iltalehti.fi 58 1,409
eenadu.net 67 1,456
newyorker.com 73 1,793
zdnet.com 76 1,831
wikihow.com 76 1,835
ign.com 62 1,993
amazon.eg 40 2,006
mashable.com 61 2,010
techradar.com 87 2,215
congress.gov 41 2,293
entrepreneur.com 89 2,421
pcmag.com 63 2,423
livescience.com 85 2,585
hindustantimes.com 90 2,672
newscientist.com 76 2,691
yumpu.com 30 2,723
chicagotribune.com 79 2,786
vanityfair.it 77 2,899
faz.net 39 2,917
today.com 62 2,969
vogue.com 82 2,989
allrecipes.com 44 3,184
letour.fr 73 3,210
myanimelist.net 70 3,276
tomshardware.com 87 3,331
lifehacker.com 65 3,342
consumerreports.org 72 3,347
amazon.se 40 3,499
amazon.ae 40 3,520

See every measured domain that blocks Meta-WebIndexer

This list is served from a cache rebuilt just now. It is discarded the moment a rescan records a policy change for this agent, so a site that has just changed its robots.txt will drop off it within one scan rather than waiting for the cache to expire.

Compare Meta

Common questions answered

Does blocking Meta-WebIndexer remove me from Meta's AI products?

Yes, in effect. Meta-WebIndexer builds the index Meta's AI products reads at question time, so disallowing it takes the site out of those answers and out of the citations that link back to it. This is the block that costs traffic.

How do I verify Meta-WebIndexer requests are genuine?

Meta's crawlers originate from AS32934, and Meta documents querying that ASN to obtain the current ranges. Map the source address to its ASN and check for AS32934. There is no published reverse DNS name.

Check your own site free

See whether your robots.txt admits Meta-WebIndexer today, and whether your edge agrees with it.