AI crawler · registry

TikTokSpider

ByteDance · Live retrieval · robots.txt token tiktokspider

ignores robotsMeasured across 29,991 sites with a robots.txt policy record for this agent, out of 29,996 on record.
The measurement

0.8% of 29,991 measured domains disallow tiktokspider in robots.txt at the site root. Separately, and independently of robots.txt, 17.0% of 29,996 reachable domains refuse an AI crawler user agent at the network edge on two consecutive probes from different locations - a refusal the site's own robots.txt does not declare and its owner often has not chosen.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "TikTokSpider blocking rate", measured 2026-11-21. https://crawlcensus.com/bot/tiktokspider

0.8%
of measured sites block it
247
sites blocking
29,744
sites allowing
no
honors robots.txt

What it is definition

Fetches shared URLs for TikTok link previews and feeds; not expected to follow robots.txt.

TikTokSpider is operated by ByteDance and classified here as live retrieval. These agents fetch a single URL because a user asked about it. Disallowing one means the assistant answers from an older copy or from someone else's summary instead of the live page. Several are documented as not bound by robots.txt, because the request is user-initiated rather than a crawl.

Disallowing tiktokspider stops the fetch that happens when a user asks TikTok link previews about one of your URLs. The answer still gets written, from an older copy or from a third party's description of the page, and you no longer control what it says.

ByteDance documents that this agent may fetch pages regardless of robots.txt. A disallow is recorded here because it states intent, but it is not the control.
robots.txt token
tiktokspider
Operator
ByteDance
Purpose
Live retrieval
Honors robots.txt
no
Documentation
https://knownagents.com/agents/tiktokspider

Identity on the wire exact string

User-agent string
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com)
Verify your own edge is not refusing it
curl -A 'Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com)' https://example.com/

Replace the host with your own. A 403, a challenge page or an empty body means the edge is blocking the agent whatever robots.txt says.

How to allow it, how to block it robots.txt

Allow it
User-agent: tiktokspider
Allow: /

Absent any rule, the agent is already allowed, so this group only matters when your * group disallows something. Naming the agent replaces the * group for it entirely, so repeat here every disallow you still want to apply to it.

Block it
User-agent: tiktokspider
Disallow: /

Disallowing tiktokspider stops the fetch that happens when a user asks TikTok link previews about one of your URLs. The answer still gets written, from an older copy or from a third party's description of the page, and you no longer control what it says.

robots.txt is not sufficient for this agent. Block by ASN at the edge. The token is not guaranteed to be present on every fetch, so treat a user-agent match as the first line only and deny the ByteDance ASN the requests actually arrive from in your logs.

Sites that block it 60 shown

DomainAI access scoreRank
msn.com 41 62
launchpad.net 52 260
telegraph.co.uk 52 580
dailymail.co.uk 61 627
dailymail.com 61 681
bild.de 81 763
theverge.com 77 815
theconversation.com 62 1,019
bfmtv.com 81 1,073
nu.nl 49 1,191
furaffinity.net 39 1,223
f95zone.to 59 1,300
wikihow.com 76 1,835
jamanetwork.com 46 1,899
themoviedb.org 74 1,984
congress.gov 41 2,293
nrk.no 69 2,325
newscientist.com 76 2,691
yumpu.com 30 2,723
imageshack.us 71 2,975
metro.co.uk 59 3,198
myanimelist.net 70 3,276
nymag.com 74 3,371
thefreedictionary.com 71 3,433
telstra.com 88 3,457
postman.com 92 3,463
digitaltrends.com 49 3,486
orf.at 69 3,542
politico.eu 62 3,595
seekingalpha.com 91 4,065
kompas.com 77 4,161
sbs.com.au 83 4,181
boredpanda.com 87 4,759
imageshack.com 71 5,037
depositphotos.com 63 5,272
dn.se 60 5,678
theepochtimes.com 80 5,684
rtl.nl 47 5,731
azcentral.com 74 6,089
sbnation.com 69 6,228
vulture.com 72 6,288
tv2.dk 74 6,352
freep.com 73 6,356
ici.fr 87 6,361
telstra.com.au 86 6,504
telegram.com 73 6,512
eater.com 59 6,522
adda247.com 88 6,537
nicematin.com 77 6,948
bandsintown.com 59 7,059
nrc.nl 84 7,106
hani.co.kr 67 7,127
game8.jp 72 7,201
gala.de 76 7,239
nzz.ch 76 7,268
15min.lt 74 7,372
nettavisen.no 63 7,562
popsugar.com 75 7,856
refinery29.com 78 7,863
nefisyemektarifleri.com 76 8,094

See every measured domain that blocks TikTokSpider

This list is served from a cache rebuilt just now. It is discarded the moment a rescan records a policy change for this agent, so a site that has just changed its robots.txt will drop off it within one scan rather than waiting for the cache to expire.

Compare ByteDance

ByteDance runs 2 agents in this registry, and they do different jobs. Blocking one says nothing about the others.

Common questions answered

Does blocking TikTokSpider remove me from TikTok link previews?

Partly. TikTokSpider fetches a page only when a user asks about it, so disallowing it does not delist you from TikTok link previews. It does mean the answer is composed without your live page, from a stale copy or a third-party summary. ByteDance documents this agent as not bound by robots.txt, so the disallow is a request rather than a control; see the blocking section for the edge rule that is.

How do I verify TikTokSpider requests are genuine?

ByteDance publishes neither reverse DNS names nor address ranges for TikTokSpider, so a request carrying this user agent cannot be authenticated. The string is trivially spoofed and is routinely reused by unrelated scrapers. Treat it as a hint, rate-limit by source address, and do not grant it access you would not grant an anonymous client.

Check your own site free

See whether your robots.txt admits TikTokSpider today, and whether your edge agrees with it.