AI access report
trustwallet.com
Open to answer engines, blocks 1 training crawlers, publishes llms.txt.
1 failing
2 partial
24 passing
Scanned 2 hours ago · 3 scans on record
trend
Reach
39.8
/ 40
Whether AI crawlers and answer-engine fetchers are permitted to request the page at all. What happens on the wire outweighs what the policy file says: a live refusal costs 16 of these 40 points, twice what the heaviest robots.txt rule costs, because a page a crawler cannot fetch is unreadable no matter how well it is marked up.
Readability
21.7
/ 25
Whether a fetcher that does not execute JavaScript receives the actual content, in markup an extractor can segment.
Structure
20
/ 20
Machine-readable markup that states the page's type, entities, canonical URL, and discrete facts instead of leaving them to be inferred.
Attribution
14
/ 15
Signals that let an answer engine name the author, date the content, resolve the publisher, and cite it under known terms.
Who is allowed to read this site 0 of 24 answer engines blocked
| Crawler | Operator | Uses content for | robots.txt | Live request |
|---|---|---|---|---|
| GPTBot Crawls content that may be used to train OpenAI's generative AI foundation models. |
OpenAI | Model training | allowed Allow: / |
served 200 |
| OAI-SearchBot Indexes pages so they can be surfaced and cited in ChatGPT search results, not for training. |
OpenAI | Answer index | allowed Allow: / |
served 200 |
| ChatGPT-User Fetches a page when a ChatGPT user or GPT Action asks for it; user-initiated, so robots rules may not apply. |
OpenAI | Live retrieval | allowed Allow: / |
not probed |
| OAI-AdsBot Visits pages submitted as ChatGPT ads to check policy compliance and ad relevance; not used for model training. |
OpenAI | Live retrieval | allowed Allow: / |
not probed |
| ClaudeBot Collects web content that may contribute to training Anthropic's models; honors Crawl-delay. |
Anthropic | Model training | allowed Allow: / |
served 200 |
| Claude-User Retrieves pages on demand when a Claude user's question needs live web content. |
Anthropic | Live retrieval | allowed Allow: / |
not probed |
| Claude-SearchBot Indexes content to improve the relevance and accuracy of Claude's search results. |
Anthropic | Answer index | allowed Allow: / |
not probed |
| anthropic-ai Legacy token widely blocked for Anthropic training; Anthropic now documents ClaudeBot, Claude-User and Claude-SearchBot. |
Anthropic | Model training | allowed Allow: / |
not probed |
| Google-Extended Control token with no user agent of its own; governs Gemini training and grounding use of Googlebot data. |
Model training | allowed Allow: / |
not probed | |
| Googlebot Crawls and renders pages for Google Search, Images, Video, News and Discover. |
Answer index | allowed Allow: / |
not probed | |
| Googlebot-News Robots token controlling Google News inclusion; crawling itself uses the Googlebot user agents. |
Answer index | allowed Allow: / |
not probed | |
| Google-CloudVertexBot Crawls sites at a site owner's request to build Vertex AI agents; no effect on Google Search. |
Live retrieval | allowed Allow: / |
not probed | |
| GoogleOther Generic Google crawler used by product teams for one-off fetches such as internal research and development. |
Model training | allowed Allow: / |
not probed | |
| Applebot Crawls for Siri, Spotlight and Safari search; falls back to Googlebot rules and ignores Crawl-delay. |
Apple | Answer index | allowed Allow: / |
not probed |
| Applebot-Extended Control token with no user agent; disallowing it excludes crawled content from Apple foundation model training. |
Apple | Model training | allowed Allow: / |
not probed |
| Bingbot Indexes pages for Bing search and the Copilot answers that are grounded in the Bing index. |
Microsoft | Answer index | allowed Allow: / |
not probed |
| msnbot Legacy Microsoft search crawler token still honored alongside bingbot. |
Microsoft | Answer index | allowed Allow: / |
not probed |
| PerplexityBot Indexes and links pages in Perplexity search results; not used to collect foundation model training data. |
Perplexity | Answer index | allowed Allow: / |
served 200 |
| Perplexity-User ignores robots Fetches a page for a specific user question; Perplexity documents that it generally ignores robots.txt. |
Perplexity | Live retrieval | allowed Allow: / |
not probed |
| Meta-ExternalAgent Crawls the web to train Meta's foundation AI models and to index content directly into products. |
Meta | Model training | allowed Allow: / |
not probed |
| Meta-ExternalFetcher ignores robots Fetches individual links for agentic AI tasks; Meta documents that it may bypass robots.txt. |
Meta | Live retrieval | allowed Allow: / |
not probed |
| FacebookBot Crawls public pages to improve language models behind Meta's speech recognition technology. |
Meta | Model training | allowed Allow: / |
not probed |
| Meta-WebIndexer Indexes pages so Meta AI can cite and link them in its search answers. |
Meta | Answer index | allowed Allow: / |
not probed |
| Meta-ExternalAds Crawls the web to improve Meta's advertising and other business products and services. |
Meta | Model training | allowed Allow: / |
not probed |
| facebookexternalhit ignores robots Fetches shared links for Facebook, Instagram and Messenger previews; may bypass robots.txt for integrity checks. |
Meta | Live retrieval | allowed Allow: / |
not probed |
| Bytespider ignores robots Downloads content to train ByteDance LLMs and is widely reported to ignore robots.txt directives. |
ByteDance | Model training | blocked Disallow: / |
not probed |
| TikTokSpider ignores robots Fetches shared URLs for TikTok link previews and feeds; not expected to follow robots.txt. |
ByteDance | Live retrieval | allowed Allow: / |
not probed |
| Amazonbot Crawls for Amazon product and Alexa answers and may use the content to train Amazon AI models. |
Amazon | Model training | allowed Allow: / |
not probed |
| Amzn-SearchBot Indexes content for Amazon search experiences such as Alexa; does not crawl for generative AI training. |
Amazon | Answer index | allowed Allow: / |
not probed |
| Amzn-User ignores robots Fetches live pages to answer a user's Alexa question; Amazon documents it may not follow all robots.txt rules. |
Amazon | Live retrieval | allowed Allow: / |
not probed |
| CCBot Builds the open Common Crawl web archive, a common source of LLM pretraining corpora. |
Common Crawl Foundation | Archive | allowed Allow: / |
not probed |
| Diffbot Extracts structured page data for Diffbot's knowledge graph, which is licensed to AI customers. |
Diffbot | Model training | allowed Allow: / |
not probed |
| omgili Collects forum, news and blog content that Webz.io sells as web data feeds, including for AI training. |
Webz.io | Model training | allowed Allow: / |
not probed |
| omgilibot Legacy Omgili search crawler token still blocked alongside the current omgili agent. |
Webz.io | Model training | allowed Allow: / |
not probed |
| AI2Bot Collects web text for Ai2's open datasets used to train open language models such as OLMo. |
Allen Institute for AI | Model training | allowed Allow: / |
not probed |
| cohere-ai Retrieves pages to answer user-initiated prompts in Cohere's enterprise AI products. |
Cohere | Live retrieval | allowed Allow: / |
not probed |
| cohere-training-data-crawler Downloads training data for the large language models behind Cohere's enterprise AI products. |
Cohere | Model training | allowed Allow: / |
not probed |
| MistralAI-User Fetches pages on demand so Mistral's Vibe assistant can answer a question with live, cited web content. |
Mistral AI | Live retrieval | allowed Allow: / |
not probed |
| MistralAI-Index Indexes content for Mistral search behind Vibe answers; not used for generative AI training. |
Mistral AI | Answer index | allowed Allow: / |
not probed |
| MistralAI-Training Crawls web content to build datasets for training Mistral's generative AI models. |
Mistral AI | Model training | allowed Allow: / |
not probed |
| DuckAssistBot Crawls pages in real time for DuckDuckGo's cited AI-assisted answers; not used for model training. |
DuckDuckGo | Live retrieval | allowed Allow: / |
not probed |
| YouBot Indexes pages for You.com search results and the AI answers built on that index. |
You.com | Answer index | allowed Allow: / |
not probed |
| PanguBot Collects web content used to train Huawei's PanGu family of large models. |
Huawei | Model training | allowed Allow: / |
not probed |
| Timpibot Crawls pages for Timpi's decentralized index, which is also used as LLM training data. |
Timpi | Model training | allowed Allow: / |
not probed |
| ImagesiftBot Downloads public images plus surrounding text to build ImageSift's searchable image index. |
ImageSift (Hive) | Model training | allowed Allow: / |
not probed |
| Kangaroo Bot Scrapes site content into datasets used to train the Kangaroo LLM. |
Kangaroo LLM | Model training | allowed Allow: / |
not probed |
| SemrushBot-OCOB Crawls pages to feed Semrush's ContentShake AI writing tool. |
Semrush | Model training | allowed Allow: / |
not probed |
| Scrapy Generic scraping framework often used to build AI training datasets; obeys robots.txt only when ROBOTSTXT_OBEY is on. |
Zyte (open-source framework) | Model training | allowed Allow: / |
not probed |
Two of these columns matter differently. robots.txt is what the site declares. Live request is what actually happened when we sent a real request using that crawler's user agent from a datacentre IP, which is how edge blocking, rate limits and challenge pages show up even when robots.txt looks permissive.
Reach 39.8 / 40
!
Training crawlers are disallowed for this path
Blocked by robots.txt: Bytespider.
Why it matters. GPTBot, ClaudeBot, and CCBot collect pages into pretraining corpora, and Google-Extended and Applebot-Extended are opt-out tokens that govern whether already-crawled pages may be used for Gemini and Apple Intelligence. Disallowing them removes your text from the corpora models generalise from, which is a policy choice, not a bug.
Fix. Decide this deliberately rather than by inheriting a template. If you want your content in model weights, remove the `Disallow: /` groups for `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, and `Applebot-Extended`. If you do not, keep those groups but scope them to the paths that matter and leave retrieval fetchers untouched, because `Google-Extended` and `Applebot-Extended` only control training use and never affect search or answer citation. Record the decision somewhere durable so the next robots.txt edit does not silently reverse it.
Fix. Decide this deliberately rather than by inheriting a template. If you want your content in model weights, remove the `Disallow: /` groups for `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, and `Applebot-Extended`. If you do not, keep those groups but scope them to the paths that matter and leave retrieval fetchers untouched, because `Google-Extended` and `Applebot-Extended` only control training use and never affect search or answer citation. Record the decision somewhere durable so the next robots.txt edit does not silently reverse it.
# Opt out of model training only.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /5 pt
✓
AI user agents receive the same 200 response as browsers
Live requests as 4 AI user agents were served normally.
16 pt
✓
Answer-engine fetchers are allowed to retrieve and cite this page
All 24 answer-engine fetchers are allowed to retrieve pages for citation.
8 pt
✓
No blanket disallow applies to this path
The wildcard group does not disallow the entire site.
4 pt
✓
robots.txt served as plain text with a 200 response
robots.txt served, 3689 bytes, 41 group(s).
2 pt
✓
Page is indexable, with no noindex directive
No noindex directive on the homepage.
2 pt
✓
No Crawl-delay directive constrains fetchers
No Crawl-delay directive.
1 pt
✓
Full-length snippet extraction is permitted
Snippets are not restricted by meta tags.
1 pt
✓
X-Robots-Tag header is absent or permissive
No restrictive X-Robots-Tag header.
1 pt
Readability 21.7 / 25
✗
Visible text is a small fraction of the HTML payload
Text is 1.4% of the 557 KB document; 206 KB is inline script.
Why it matters. Extractors strip scripts, styles, and wrapper markup before passing text to a model, and a page where content is a small fraction of the payload loses more of it to boilerplate removal. Inline JSON state blobs and deeply nested wrappers also push real text past the truncation limit of a retrieval context.
Fix. Move inline hydration state and large inline scripts out of the document, or fetch them after load instead of embedding them. Flatten wrapper `div` trees and let semantic elements carry the content, and keep utility-class soup out of the article body. Serve the same text without the boilerplate at a stable URL if you need a clean extraction target.
Fix. Move inline hydration state and large inline scripts out of the document, or fetch them after load instead of embedding them. Flatten wrapper `div` trees and let semantic elements carry the content, and keep utility-class soup out of the article body. Serve the same text without the boilerplate at a stable URL if you need a clean extraction target.
4 pt
✓
Substantive text is present in the server-rendered HTML
1244 words of text are present in the raw HTML. Most AI fetchers do not run JavaScript.
9 pt
✓
Primary content is wrapped in a semantic landmark
A <main> landmark marks the primary content.
3 pt
✓
Headings form a single, ordered outline
1 H1 and 26 headings total.
3 pt
✓
Title is unique and describes the page in specific terms
Title is 57 characters: "Best Crypto Wallet for Web3, NFTs and DeFi | Trust Wallet"
3 pt
✓
Meta description provides an author-written summary
Meta description is 159 characters.
2 pt
✓
Document language is declared on the html element
Declared language: en.
1 pt
Structure 20 / 20
✓
Page ships JSON-LD structured data
39 JSON-LD node(s): Organization, ImageObject, ItemList, SiteNavigationElement, WebSite, SearchAction.
7 pt
✓
Structured data uses a specific type that matches the page
Recognized types: organization, website, softwareapplication, faqpage.
4 pt
✓
JSON-LD parses cleanly with recognised schema.org terms
All JSON-LD blocks parse cleanly.
3 pt
✓
Page declares a self-referential canonical URL
Canonical: https://trustwallet.com/
3 pt
✓
XML sitemap is declared in robots.txt and returns 200
Sitemap found at /sitemap.xml (4 URLs on the first document).
2 pt
✓
Key facts are available in lists or tables
0 tables, 5 lists, 0 code blocks, 0 question headings.
1 pt
Attribution 14 / 15
!
No machine-readable license or usage terms for the content
No licence declaration, so reuse terms are ambiguous.
Why it matters. A `license` property, or a linked terms page, states the reuse conditions in a place a crawler can read, rather than leaving them to be inferred. Where terms are unstated, some pipelines default to the more restrictive handling, which reduces how much of the text is quoted.
Fix. Add a `license` property to the page's structured data pointing at a specific license URL, such as a Creative Commons deed or your own terms page, and add `rel="license"` on the visible link. Use `usageInfo` for conditions that are not a standard license, such as attribution wording or an API-only clause. State the terms once, at a stable URL, and reference it from every page rather than restating it per template.
Fix. Add a `license` property to the page's structured data pointing at a specific license URL, such as a Creative Commons deed or your own terms page, and add `rel="license"` on the visible link. Use `usageInfo` for conditions that are not a standard license, such as attribution wording or an API-only clause. State the terms once, at a stable URL, and reference it from every page rather than restating it per template.
<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Allow AI answer engines in robots.txt",
"license": "https://creativecommons.org/licenses/by/4.0/",
"usageInfo": "https://example.com/legal/content-reuse",
"creditText": "Example, crawlcensus research desk"
}
</script>2 pt
✓
/llms.txt lists canonical pages in the documented format
/llms.txt is present and well formed (13 sections, 328 links).
4 pt
✓
Author is declared in structured data
Authorship is declared in structured data or markup.
3 pt
✓
Published and modified dates are declared in ISO 8601
dateModified is published.
3 pt
✓
Organization entity declares the publisher and its identifiers
Organization entity with sameAs links is present.
3 pt
Evidence raw measurements
robots.txt
- Size
- 3,689 bytes
- Groups
- 41
- Sitemaps
- https://trustwallet.com/sitemap.xml
View the file as our crawler received it
User-agent: *
Allow: /
Disallow: /api/
Disallow: /landing/mobile-application.mp4
# _rsc stays blocked: Next.js internal payloads, not HTML pages (crawl trap).
Disallow: /*?_rsc=*
# Tracking params (utm_, gclid, fbclid, ...) are deliberately NOT blocked:
# those URLs serve the full page with a canonical tag, so Google must be able
# to crawl them to consolidate signals into the clean URL. Blocking them made
# Google index /?utm_source=... as a separate page (GSC top page, 1M clicks).
# ------------------------------------------------------------
# Search engine crawlers
# ------------------------------------------------------------
User-agent: Googlebot
Allow: /
User-agent: GoogleOther
Allow: /
User-agent: Google-CloudVertexBot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: DuckDuckBot
Allow: /
User-agent: Baiduspider
Allow: /
User-agent: Yeti
Allow: /
# ------------------------------------------------------------
# AI / LLM crawlers — training, indexing, real-time citation
# ------------------------------------------------------------
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Legacy Anthropic token (kept as no-op for historical compatibility)
User-agent: anthropic-ai
Allow: /
# Google AI training opt-in (Gemini, Vertex)
User-agent: Google-Extended
Allow: /
# Perplexity (PerplexityBot=index, Perplexity-User=user-triggered real-time)
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Apple (Applebot=Siri/Spotlight/Safari, Applebot-Extended=Apple Intelligence training)
User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Allow: /
# Meta (training + user-triggered + indexer + social unfurl)
User-agent: Meta-ExternalAgent
Allow: /
User-agent: Meta-ExternalFetcher
Allow: /
User-agent: meta-webindexer
Allow: /
# Mistral Le Chat real-time
User-agent: MistralAI-User
Allow: /
# Amazon (Alexa + Amazon AI)
User-agent: Amazonbot
Allow: /
# Common Crawl (open web corpus, ingested by many LLMs)
User-agent: CCBot
Allow: /
# DuckDuckGo AI answers
User-agent: DuckAssistBot
Allow: /
# You.com AI search
User-agent: YouBot
Allow: /
# Diffbot (knowledge graph used by many enterprise AI buyers)
User-agent: Diffbot
Allow: /
# Webz.io / Omgili (forum index licensed to AI buyers)
User-agent: Omgilibot
Allow: /
# ------------------------------------------------------------
# Adversarial AI crawler — block
# ------------------------------------------------------------
# ByteDance Bytespider — compliance unreliable per multiple reports.
# Also recommended to block at WAF layer.
User-agent: Bytespider
Disallow: /
# ------------------------------------------------------------
# Social link-preview / unfurl bots
# ------------------------------------------------------------
User-agent: facebookexternalhit
Allow: /
User-agent: Twitterbot
Allow: /
User-agent: LinkedInBot
Allow: /
User-agent: Slackbot
Allow: /
User-agent: Discordbot
Allow: /
User-agent: TelegramBot
Allow: /
User-agent: WhatsApp
Allow: /
# ------------------------------------------------------------
# SEO research tools (kept for our own auditing)
# ------------------------------------------------------------
User-agent: AhrefsBot
Allow: /
User-agent: AhrefsSiteAudit
Allow: /
User-agent: Yandex
Allow: /
Clean-param: utm_source&utm_medium&utm_campaign&utm_term&utm_content
Clean-param: gclid&fbclid&yclid&msclkid&ttclid
Clean-param: ref&ref_src&ref_url&referrer
Clean-param: mc_cid&mc_eid
Clean-param: _gl&_ga
Sitemap: https://trustwallet.com/sitemap.xml
Machine-readable extras
- llms.txt
- valid, 71,800 bytes, 328 links
- ai.txt
- absent
- Sitemap
- /sitemap.xml (4 URLs)
- Feeds
- none
- Schema types
- Organization, ImageObject, ItemList, SiteNavigationElement, WebSite, SearchAction, EntryPoint, SoftwareApplication, Offer, AggregateRating
- Edge
- not identified
- Final URL
- https://trustwallet.com/
- HTML size
- 557 KB, 1,244 words of text
Compare it with a competitor head to head
Put trustwallet.com next to another domain and see which one an answer engine can actually read. Both sides are measured the same way, so the difference is the finding.
Already measured: vs langchain.com · vs walkscore.com · vs 24-hds.com
Publish the score free
Sites that score well embed the badge. It links back to this live report, which re-measures on every scan.
<a href="https://crawlcensus.com/site/trustwallet.com"><img src="https://crawlcensus.com/badge/trustwallet.com.svg" alt="AI access score for trustwallet.com" width="150" height="20"></a>Track changes on this domain free for one
Crawler policy is edited quietly. We re-scan monitored domains daily, keep the history, and email you the moment a crawler is blocked or unblocked, an llms.txt appears, or the score moves.
Free for one domain per address, and always free for domains you have verified you own. No newsletter, only change alerts.
Pro follows 25 domains, keeps 90 days of history, compares them against each other and exports the lot as CSV, for 19 dollars a month. That is the difference: measuring is free, watching a portfolio is not.