ByteDance
Everything this census measures about the crawlers ByteDance runs against the public web, in one place. Measured by direct request across 30,321 domains, not inferred from a third-party index.
ByteDance runs 2 crawlers tracked by this census. Its most-disallowed agent is Bytespider, disallowed in robots.txt by 11% of the 30,317 domains that served this crawler a page. A site that refuses every programmatic client is excluded from that denominator, because its refusal says nothing about AI policy; the deny list is longer than this figure implies, and states why in its own header.
Measured 2026-08-24 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "ByteDance crawler access", measured 2026-08-24. https://crawlcensus.com/operator/bytedance
Every agent we track 2 for ByteDance
| Agent | Purpose | Honours robots.txt | Disallowed in robots.txt | Refused at the edge |
|---|---|---|---|---|
Bytespiderbytespider |
model training | operator says no | 10.6% 3,207 of 30,317 |
not probed |
TikTokSpidertiktokspider |
live retrieval | operator says no | 0.8% 244 of 30,317 |
not probed |
Agents documented as not bound by robots.txt 2 of 2
ByteDance's own documentation states that Bytespider, TikTokSpider may fetch a page regardless of robots.txt, typically because the fetch is user-initiated. A disallow is still recorded here because it states the site's intent, but it is not a control, and nothing measured from outside can verify compliance either way.
If that characterisation is out of date, it is exactly the kind of thing the correction channel below is for.
Right of reply how to correct this page
This page makes measured claims about ByteDance's products. Three kinds of thing can be wrong with it, and all three are worth fixing:
- The registry is stale
- A token, user-agent string, documentation URL or stated purpose has changed, or an agent has been retired or added.
- The classification is wrong
- An agent is recorded as training when it is retrieval, or as ignoring robots.txt when it now honours it.
- A measurement looks wrong
- A specific figure does not match what you observe from your side. Say which, and the raw record for every domain behind it is in the open dataset.
The correction channel is deliberately boring: send corrections to hello@crawlcensus.com from a domain you control, or open an issue against the public registry. Corrections to factual registry data are applied without argument; disputed measurements get the underlying scan records published alongside the dispute rather than quietly amended.
The method is published in full, every figure is reproducible from the open dataset, and the scoring model is versioned so an old citation stays interpretable.
If you operate these crawlers the useful endpoint
The measurement behind this page is also available as a preflight check, so your crawler can find out whether a domain will actually serve it before spending the request.
curl -X POST https://crawlcensus.com/api/v1/preflight \
-H 'content-type: application/json' \
-d '{"agent":"bytespider","domains":["example.com"]}'Also exposed over MCP as crawl_preflight and agent_profile at https://crawlcensus.com/mcp. Details at /for-crawlers.