AI crawler · registry

facebookexternalhit

Meta · Live retrieval · robots.txt token facebookexternalhit

ignores robotsMeasured across 29,991 sites with a robots.txt policy record for this agent, out of 29,996 on record.
The measurement

0.2% of 29,991 measured domains disallow facebookexternalhit in robots.txt at the site root. Separately, and independently of robots.txt, 17.0% of 29,996 reachable domains refuse an AI crawler user agent at the network edge on two consecutive probes from different locations - a refusal the site's own robots.txt does not declare and its owner often has not chosen.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "facebookexternalhit blocking rate", measured 2026-11-21. https://crawlcensus.com/bot/facebookexternalhit

0.2%
of measured sites block it
56
sites blocking
29,935
sites allowing
no
honors robots.txt

What it is definition

Fetches shared links for Facebook, Instagram and Messenger previews; may bypass robots.txt for integrity checks.

facebookexternalhit is operated by Meta and classified here as live retrieval. These agents fetch a single URL because a user asked about it. Disallowing one means the assistant answers from an older copy or from someone else's summary instead of the live page. Several are documented as not bound by robots.txt, because the request is user-initiated rather than a crawl.

Disallowing facebookexternalhit stops the fetch that happens when a user asks Meta's AI products about one of your URLs. The answer still gets written, from an older copy or from a third party's description of the page, and you no longer control what it says.

Meta documents that this agent may fetch pages regardless of robots.txt. A disallow is recorded here because it states intent, but it is not the control.
robots.txt token
facebookexternalhit
Operator
Meta
Purpose
Live retrieval
Honors robots.txt
no
Documentation
https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/

Identity on the wire exact string

User-agent string
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
Verify your own edge is not refusing it
curl -A 'facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)' https://example.com/

Replace the host with your own. A 403, a challenge page or an empty body means the edge is blocking the agent whatever robots.txt says.

How to allow it, how to block it robots.txt

Allow it
User-agent: facebookexternalhit
Allow: /

Absent any rule, the agent is already allowed, so this group only matters when your * group disallows something. Naming the agent replaces the * group for it entirely, so repeat here every disallow you still want to apply to it.

Block it
User-agent: facebookexternalhit
Disallow: /

Disallowing facebookexternalhit stops the fetch that happens when a user asks Meta's AI products about one of your URLs. The answer still gets written, from an older copy or from a third party's description of the page, and you no longer control what it says.

robots.txt is not sufficient for this agent. Block it at the edge on the user-agent string, since Meta does not treat the file as binding.

Sites that block it 60 shown

DomainAI access scoreRank
leboncoin.fr 55 1,187
furaffinity.net 39 1,223
wikihow.com 76 1,835
congress.gov 41 2,293
faz.net 39 2,917
myanimelist.net 70 3,276
flipboard.com 64 3,286
politico.eu 62 3,595
codeberg.org 41 3,868
nationalpost.com 71 5,681
nicematin.com 77 6,948
kufar.by 59 7,580
refinery29.com 78 7,863
ylilauta.org 50 9,842
sgxnifty.org 91 10,093
financialpost.com 71 10,495
gamblingtherapy.org 87 11,504
vancouversun.com 71 13,419
torontosun.com 71 13,668
ilex.by 59 13,871
canada.com 71 14,862
thieme-connect.com 39 15,233
hinative.com 40 16,052
page-stats.de 49 16,122
lightningmaps.org 51 16,664
ina.fr 76 17,492
locally.com 88 18,306
ottawacitizen.com 71 19,941
supersport.hr 51 20,318
calgaryherald.com 71 21,685
desy.de 62 21,823
zin.ru 41 24,525
pecom.ru 95 25,811
straightdope.com 76 26,342
texasattorneygeneral.gov 65 27,737
u-cursos.cl 51 29,139
ulife.com.br 54 29,224
edmontonjournal.com 71 29,639
thieme-connect.de 79 29,701
bsc.es 37 30,387
vndb.org 60 31,282
ticketspice.com 92 31,768
zumper.com 78 34,581
saltwire.com 70 36,237
ytmnd.com 41 36,334
eurobilltracker.com 65 36,372
igt.com 48 38,409
regfox.com 81 40,628
chipsmall.com 82 41,232
gunbark.dev 37 41,368
oefb.at 50 41,542
iex.nl 48 42,076
horseracingnation.com 72 42,862
gipuzkoa.eus 70 43,909
teambelgium.net 59 43,983
theprovince.com 71 44,299
tupalo.com 57 45,390
wikihow.life 76 46,635
edurev.in 89 46,671
alisweb.org 39 46,923

See every measured domain that blocks facebookexternalhit

This list is served from a cache rebuilt just now. It is discarded the moment a rescan records a policy change for this agent, so a site that has just changed its robots.txt will drop off it within one scan rather than waiting for the cache to expire.

Compare Meta

Common questions answered

Does blocking facebookexternalhit remove me from Meta's AI products?

Partly. facebookexternalhit fetches a page only when a user asks about it, so disallowing it does not delist you from Meta's AI products. It does mean the answer is composed without your live page, from a stale copy or a third-party summary. Meta documents this agent as not bound by robots.txt, so the disallow is a request rather than a control; see the blocking section for the edge rule that is.

How do I verify facebookexternalhit requests are genuine?

Meta's crawlers originate from AS32934, and Meta documents querying that ASN to obtain the current ranges. Map the source address to its ASN and check for AS32934. There is no published reverse DNS name.

Check your own site free

See whether your robots.txt admits facebookexternalhit today, and whether your edge agrees with it.