Zyte (open-source framework)
Everything this census measures about the crawlers Zyte (open-source framework) runs against the public web, in one place. Measured by direct request across 29,465 domains, not inferred from a third-party index.
Zyte (open-source framework) runs 1 crawler tracked by this census. Its most-disallowed agent is Scrapy, disallowed in robots.txt by 2% of the 29,120 domains that served this crawler a page. A site that refuses every programmatic client is excluded from that denominator, because its refusal says nothing about AI policy; the deny list is longer than this figure implies, and states why in its own header.
Measured 2026-08-24 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "Zyte (open-source framework) crawler access", measured 2026-08-24. https://crawlcensus.com/operator/zyte-open-source-framework
Every agent we track 1 for Zyte (open-source framework)
| Agent | Purpose | Honours robots.txt | Disallowed in robots.txt | Refused at the edge |
|---|---|---|---|---|
Scrapyscrapy |
model training | yes | 2.3% 669 of 29,120 |
not probed |
Right of reply how to correct this page
This page makes measured claims about Zyte (open-source framework)'s products. Three kinds of thing can be wrong with it, and all three are worth fixing:
- The registry is stale
- A token, user-agent string, documentation URL or stated purpose has changed, or an agent has been retired or added.
- The classification is wrong
- An agent is recorded as training when it is retrieval, or as ignoring robots.txt when it now honours it.
- A measurement looks wrong
- A specific figure does not match what you observe from your side. Say which, and the raw record for every domain behind it is in the open dataset.
The correction channel is deliberately boring: send corrections to hello@crawlcensus.com from a domain you control, or open an issue against the public registry. Corrections to factual registry data are applied without argument; disputed measurements get the underlying scan records published alongside the dispute rather than quietly amended.
The method is published in full, every figure is reproducible from the open dataset, and the scoring model is versioned so an old citation stays interpretable.
If you operate these crawlers the useful endpoint
The measurement behind this page is also available as a preflight check, so your crawler can find out whether a domain will actually serve it before spending the request.
curl -X POST https://crawlcensus.com/api/v1/preflight \
-H 'content-type: application/json' \
-d '{"agent":"scrapy","domains":["example.com"]}'Also exposed over MCP as crawl_preflight and agent_profile at https://crawlcensus.com/mcp. Details at /for-crawlers.