Meta
Everything this census measures about the crawlers Meta runs against the public web, in one place. Measured by direct request across 29,465 domains, not inferred from a third-party index.
Meta runs 6 crawlers tracked by this census. Its most-disallowed agent is Meta-ExternalAgent, disallowed in robots.txt by 9% of the 29,120 domains that served this crawler a page. A site that refuses every programmatic client is excluded from that denominator, because its refusal says nothing about AI policy; the deny list is longer than this figure implies, and states why in its own header.
Measured 2026-08-24 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "Meta crawler access", measured 2026-08-24. https://crawlcensus.com/operator/meta
Every agent we track 6 for Meta
| Agent | Purpose | Honours robots.txt | Disallowed in robots.txt | Refused at the edge |
|---|---|---|---|---|
Meta-ExternalAgentmeta-externalagent |
model training | yes | 9.0% 2,607 of 29,120 |
not probed |
FacebookBotfacebookbot |
model training | yes | 3.2% 940 of 29,120 |
not probed |
Meta-ExternalFetchermeta-externalfetcher |
live retrieval | operator says no | 1.7% 501 of 29,120 |
not probed |
Meta-WebIndexermeta-webindexer |
answer index | yes | 1.1% 323 of 29,120 |
not probed |
facebookexternalhitfacebookexternalhit |
live retrieval | operator says no | 0.2% 56 of 29,120 |
not probed |
Meta-ExternalAdsmeta-externalads |
model training | yes | 0.1% 17 of 29,120 |
not probed |
Agents documented as not bound by robots.txt 2 of 6
Meta's own documentation states that Meta-ExternalFetcher, facebookexternalhit may fetch a page regardless of robots.txt, typically because the fetch is user-initiated. A disallow is still recorded here because it states the site's intent, but it is not a control, and nothing measured from outside can verify compliance either way.
If that characterisation is out of date, it is exactly the kind of thing the correction channel below is for.
Right of reply how to correct this page
This page makes measured claims about Meta's products. Three kinds of thing can be wrong with it, and all three are worth fixing:
- The registry is stale
- A token, user-agent string, documentation URL or stated purpose has changed, or an agent has been retired or added.
- The classification is wrong
- An agent is recorded as training when it is retrieval, or as ignoring robots.txt when it now honours it.
- A measurement looks wrong
- A specific figure does not match what you observe from your side. Say which, and the raw record for every domain behind it is in the open dataset.
The correction channel is deliberately boring: send corrections to hello@crawlcensus.com from a domain you control, or open an issue against the public registry. Corrections to factual registry data are applied without argument; disputed measurements get the underlying scan records published alongside the dispute rather than quietly amended.
The method is published in full, every figure is reproducible from the open dataset, and the scoring model is versioned so an old citation stays interpretable.
If you operate these crawlers the useful endpoint
The measurement behind this page is also available as a preflight check, so your crawler can find out whether a domain will actually serve it before spending the request.
curl -X POST https://crawlcensus.com/api/v1/preflight \
-H 'content-type: application/json' \
-d '{"agent":"meta-externalagent","domains":["example.com"]}'Also exposed over MCP as crawl_preflight and agent_profile at https://crawlcensus.com/mcp. Details at /for-crawlers.