ocu.org
Open to answer engines.
Whether AI crawlers and answer-engine fetchers are permitted to request the page at all. What happens on the wire outweighs what the policy file says: a live refusal costs 16 of these 40 points, twice what the heaviest robots.txt rule costs, because a page a crawler cannot fetch is unreadable no matter how well it is marked up.
Whether a fetcher that does not execute JavaScript receives the actual content, in markup an extractor can segment.
Machine-readable markup that states the page's type, entities, canonical URL, and discrete facts instead of leaving them to be inferred.
Signals that let an answer engine name the author, date the content, resolve the publisher, and cite it under known terms.
Who is allowed to read this site 0 of 24 answer engines blocked
| Crawler | Operator | Uses content for | robots.txt | Live request |
|---|---|---|---|---|
| GPTBot Crawls content that may be used to train OpenAI's generative AI foundation models. |
OpenAI | Model training | allowed | served 200 |
| OAI-SearchBot Indexes pages so they can be surfaced and cited in ChatGPT search results, not for training. |
OpenAI | Answer index | allowed Allow: / |
served 200 |
| ChatGPT-User Fetches a page when a ChatGPT user or GPT Action asks for it; user-initiated, so robots rules may not apply. |
OpenAI | Live retrieval | allowed Allow: / |
not probed |
| OAI-AdsBot Visits pages submitted as ChatGPT ads to check policy compliance and ad relevance; not used for model training. |
OpenAI | Live retrieval | allowed | not probed |
| ClaudeBot Collects web content that may contribute to training Anthropic's models; honors Crawl-delay. |
Anthropic | Model training | allowed | served 200 |
| Claude-User Retrieves pages on demand when a Claude user's question needs live web content. |
Anthropic | Live retrieval | allowed | not probed |
| Claude-SearchBot Indexes content to improve the relevance and accuracy of Claude's search results. |
Anthropic | Answer index | allowed | not probed |
| anthropic-ai Legacy token widely blocked for Anthropic training; Anthropic now documents ClaudeBot, Claude-User and Claude-SearchBot. |
Anthropic | Model training | allowed | not probed |
| Google-Extended Control token with no user agent of its own; governs Gemini training and grounding use of Googlebot data. |
Model training | allowed | not probed | |
| Googlebot Crawls and renders pages for Google Search, Images, Video, News and Discover. |
Answer index | allowed | not probed | |
| Googlebot-News Robots token controlling Google News inclusion; crawling itself uses the Googlebot user agents. |
Answer index | allowed | not probed | |
| Google-CloudVertexBot Crawls sites at a site owner's request to build Vertex AI agents; no effect on Google Search. |
Live retrieval | allowed | not probed | |
| GoogleOther Generic Google crawler used by product teams for one-off fetches such as internal research and development. |
Model training | allowed | not probed | |
| Applebot Crawls for Siri, Spotlight and Safari search; falls back to Googlebot rules and ignores Crawl-delay. |
Apple | Answer index | allowed | not probed |
| Applebot-Extended Control token with no user agent; disallowing it excludes crawled content from Apple foundation model training. |
Apple | Model training | allowed | not probed |
| Bingbot Indexes pages for Bing search and the Copilot answers that are grounded in the Bing index. |
Microsoft | Answer index | allowed | not probed |
| msnbot Legacy Microsoft search crawler token still honored alongside bingbot. |
Microsoft | Answer index | allowed | not probed |
| PerplexityBot Indexes and links pages in Perplexity search results; not used to collect foundation model training data. |
Perplexity | Answer index | allowed Allow: / |
served 200 |
| Perplexity-User ignores robots Fetches a page for a specific user question; Perplexity documents that it generally ignores robots.txt. |
Perplexity | Live retrieval | allowed | not probed |
| Meta-ExternalAgent Crawls the web to train Meta's foundation AI models and to index content directly into products. |
Meta | Model training | allowed | not probed |
| Meta-ExternalFetcher ignores robots Fetches individual links for agentic AI tasks; Meta documents that it may bypass robots.txt. |
Meta | Live retrieval | allowed | not probed |
| FacebookBot Crawls public pages to improve language models behind Meta's speech recognition technology. |
Meta | Model training | allowed | not probed |
| Meta-WebIndexer Indexes pages so Meta AI can cite and link them in its search answers. |
Meta | Answer index | allowed | not probed |
| Meta-ExternalAds Crawls the web to improve Meta's advertising and other business products and services. |
Meta | Model training | allowed | not probed |
| facebookexternalhit ignores robots Fetches shared links for Facebook, Instagram and Messenger previews; may bypass robots.txt for integrity checks. |
Meta | Live retrieval | allowed | not probed |
| Bytespider ignores robots Downloads content to train ByteDance LLMs and is widely reported to ignore robots.txt directives. |
ByteDance | Model training | allowed | not probed |
| TikTokSpider ignores robots Fetches shared URLs for TikTok link previews and feeds; not expected to follow robots.txt. |
ByteDance | Live retrieval | allowed | not probed |
| Amazonbot Crawls for Amazon product and Alexa answers and may use the content to train Amazon AI models. |
Amazon | Model training | allowed | not probed |
| Amzn-SearchBot Indexes content for Amazon search experiences such as Alexa; does not crawl for generative AI training. |
Amazon | Answer index | allowed | not probed |
| Amzn-User ignores robots Fetches live pages to answer a user's Alexa question; Amazon documents it may not follow all robots.txt rules. |
Amazon | Live retrieval | allowed | not probed |
| CCBot Builds the open Common Crawl web archive, a common source of LLM pretraining corpora. |
Common Crawl Foundation | Archive | allowed | not probed |
| Diffbot Extracts structured page data for Diffbot's knowledge graph, which is licensed to AI customers. |
Diffbot | Model training | allowed | not probed |
| omgili Collects forum, news and blog content that Webz.io sells as web data feeds, including for AI training. |
Webz.io | Model training | allowed | not probed |
| omgilibot Legacy Omgili search crawler token still blocked alongside the current omgili agent. |
Webz.io | Model training | allowed | not probed |
| AI2Bot Collects web text for Ai2's open datasets used to train open language models such as OLMo. |
Allen Institute for AI | Model training | allowed | not probed |
| cohere-ai Retrieves pages to answer user-initiated prompts in Cohere's enterprise AI products. |
Cohere | Live retrieval | allowed | not probed |
| cohere-training-data-crawler Downloads training data for the large language models behind Cohere's enterprise AI products. |
Cohere | Model training | allowed | not probed |
| MistralAI-User Fetches pages on demand so Mistral's Vibe assistant can answer a question with live, cited web content. |
Mistral AI | Live retrieval | allowed | not probed |
| MistralAI-Index Indexes content for Mistral search behind Vibe answers; not used for generative AI training. |
Mistral AI | Answer index | allowed | not probed |
| MistralAI-Training Crawls web content to build datasets for training Mistral's generative AI models. |
Mistral AI | Model training | allowed | not probed |
| DuckAssistBot Crawls pages in real time for DuckDuckGo's cited AI-assisted answers; not used for model training. |
DuckDuckGo | Live retrieval | allowed | not probed |
| YouBot Indexes pages for You.com search results and the AI answers built on that index. |
You.com | Answer index | allowed Allow: / |
not probed |
| PanguBot Collects web content used to train Huawei's PanGu family of large models. |
Huawei | Model training | allowed | not probed |
| Timpibot Crawls pages for Timpi's decentralized index, which is also used as LLM training data. |
Timpi | Model training | allowed | not probed |
| ImagesiftBot Downloads public images plus surrounding text to build ImageSift's searchable image index. |
ImageSift (Hive) | Model training | allowed | not probed |
| Kangaroo Bot Scrapes site content into datasets used to train the Kangaroo LLM. |
Kangaroo LLM | Model training | allowed | not probed |
| SemrushBot-OCOB Crawls pages to feed Semrush's ContentShake AI writing tool. |
Semrush | Model training | allowed | not probed |
| Scrapy Generic scraping framework often used to build AI training datasets; obeys robots.txt only when ROBOTSTXT_OBEY is on. |
Zyte (open-source framework) | Model training | allowed | not probed |
Reach 40 / 40
AI user agents receive the same 200 response as browsers
Live requests as 4 AI user agents were served normally.
Answer-engine fetchers are allowed to retrieve and cite this page
All 24 answer-engine fetchers are allowed to retrieve pages for citation.
Training crawlers may fetch this path
All 23 tracked training crawlers are allowed.
No blanket disallow applies to this path
The wildcard group does not disallow the entire site.
robots.txt served as plain text with a 200 response
robots.txt served, 26386 bytes, 5 group(s).
Page is indexable, with no noindex directive
No noindex directive on the homepage.
No Crawl-delay directive constrains fetchers
No Crawl-delay directive.
Full-length snippet extraction is permitted
Snippets are not restricted by meta tags.
X-Robots-Tag header is absent or permissive
No restrictive X-Robots-Tag header.
Readability 23.1 / 25
Visible text is a small fraction of the HTML payload
Text is 4.3% of the 294 KB document; 38 KB is inline script.
Fix. Move inline hydration state and large inline scripts out of the document, or fetch them after load instead of embedding them. Flatten wrapper `div` trees and let semantic elements carry the content, and keep utility-class soup out of the article body. Serve the same text without the boilerplate at a stable URL if you need a clean extraction target.
Substantive text is present in the server-rendered HTML
1983 words of text are present in the raw HTML. Most AI fetchers do not run JavaScript.
Primary content is wrapped in a semantic landmark
A <main> landmark marks the primary content.
Headings form a single, ordered outline
1 H1 and 48 headings total, 3 phrased as questions.
Title is unique and describes the page in specific terms
Title is 44 characters: "OCU. Organización de Consumidores y Usuarios"
Meta description provides an author-written summary
Meta description is 212 characters.
Document language is declared on the html element
Declared language: es-ES.
Structure 18 / 20
Structured data is too generic for what the page is about
Recognized types: organization.
Fix. Replace bare `WebPage` and `WebSite` nodes with the most specific type that describes the page, and fill the properties that type defines. Use `@graph` to publish several linked nodes on one page, such as an `Article` whose `publisher` points at an `Organization` node by `@id`. Add `BreadcrumbList` for hierarchy and reuse the same `@id` values across pages so the entity resolves to one record.
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Product",
"@id": "https://example.com/products/widget#product",
"name": "Widget Pro",
"sku": "WGT-PRO-1",
"brand": { "@type": "Brand", "name": "Example" },
"offers": {
"@type": "Offer",
"url": "https://example.com/products/widget",
"price": "49.00",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock"
}
},
{
"@type": "BreadcrumbList",
"itemListElement": [
{ "@type": "ListItem", "position": 1, "name": "Products", "item": "https://example.com/products" },
{ "@type": "ListItem", "position": 2, "name": "Widget Pro" }
]
}
]
}ReferencePage ships JSON-LD structured data
9 JSON-LD node(s): Organization, ContactPoint, PostalAddress, ImageObject, QuantitativeValue, SearchAction.
JSON-LD parses cleanly with recognised schema.org terms
All JSON-LD blocks parse cleanly.
Page declares a self-referential canonical URL
Canonical: https://www.ocu.org/
XML sitemap is declared in robots.txt and returns 200
Sitemap found at /sitemap.xml (8 URLs on the first document).
Key facts are available in lists or tables
0 tables, 46 lists, 0 code blocks, 3 question headings.
Attribution 4 / 9
No /llms.txt index of canonical pages
No /llms.txt.
Fix. Publish `/llms.txt` as `text/plain` markdown: an `#` H1 with the project name, a `>` blockquote summary, optional plain paragraphs of context, then `##` sections whose bullets are `[title](absolute-url): note`. Link the pages you want quoted, put lower-priority links under an `## Optional` section, and prefer URLs that also serve clean markdown. Keep it generated from the same source as your sitemap so it does not drift, and remember it is a hint for assistants, not an access control mechanism.
# Example
> Example publishes reference documentation for the Widget API and guides for
> configuring crawler access.
Prefer the pages below over search results; each URL is canonical.
## Docs
- [Widget API reference](https://example.com/docs/api): endpoints, auth, limits.
- [Quickstart](https://example.com/docs/quickstart): first request in five minutes.
## Policies
- [Crawler policy](https://example.com/legal/crawlers): which agents we allow.
## Optional
- [Changelog](https://example.com/changelog): dated release notes.ReferenceNo machine-readable license or usage terms for the content
No licence declaration, so reuse terms are ambiguous.
Fix. Add a `license` property to the page's structured data pointing at a specific license URL, such as a Creative Commons deed or your own terms page, and add `rel="license"` on the visible link. Use `usageInfo` for conditions that are not a standard license, such as attribution wording or an API-only clause. State the terms once, at a stable URL, and reference it from every page rather than restating it per template.
<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Allow AI answer engines in robots.txt",
"license": "https://creativecommons.org/licenses/by/4.0/",
"usageInfo": "https://example.com/legal/content-reuse",
"creditText": "Example, crawlcensus research desk"
}
</script>ReferenceNo machine-readable author is attached to the page
This page declares no article-type structured data, so a machine-readable byline is not expected of it. The check is excluded from the score rather than counted as a failure.
Fix. Add an `author` property to the page's `Article`, `BlogPosting`, or `NewsArticle` node, typed as `Person` or `Organization`, with a `name` and a `url` pointing at a real profile page. Give each author a stable `@id` and reuse it across posts so the entity consolidates. Keep the visible byline identical to the structured value, and avoid generic names such as "Admin" or "Staff Writer" where a real author exists.
{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Allow AI answer engines in robots.txt",
"author": {
"@type": "Person",
"@id": "https://example.com/authors/dana-reyes#person",
"name": "Dana Reyes",
"url": "https://example.com/authors/dana-reyes",
"jobTitle": "Infrastructure Engineer",
"sameAs": ["https://github.com/danareyes"]
}
}ReferenceNo machine-readable published or modified date
This page declares no article-type structured data, so publication and modification dates are not expected of it. The check is excluded from the score rather than counted as a failure.
Fix. Publish `datePublished` and `dateModified` in the page's structured data as ISO 8601 values with a timezone offset. Update `dateModified` only when the content actually changes, since bumping it on every deploy trains crawlers to ignore it. Mirror the value in a visible `<time datetime>` element so the rendered text and the metadata agree, and keep the sitemap `lastmod` consistent with it.
<time datetime="2026-08-04T14:20:00-04:00">Updated August 4, 2026</time>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Allow AI answer engines in robots.txt",
"datePublished": "2026-02-11T09:00:00-05:00",
"dateModified": "2026-08-04T14:20:00-04:00"
}
</script>ReferenceOrganization entity declares the publisher and its identifiers
Organization entity with sameAs links is present.
Evidence raw measurements
- Size
- 26,386 bytes
- Groups
- 5
- Sitemaps
- https://www.ocu.org/sitemap.xml
https://www.ocu.org/news-sitemap.xml
https://www.ocu.org/inversiones/sitemap.xml
https://www.ocu.org/fincas-y-casas/sitemap.xml
View the file as our crawler received it
#ALine sitemap
Sitemap: https://www.ocu.org/sitemap.xml
#Google News Sitemap
Sitemap: https://www.ocu.org/news-sitemap.xml
#FLine sitemaps
Sitemap: https://www.ocu.org/inversiones/sitemap.xml
Sitemap: https://www.ocu.org/fincas-y-casas/sitemap.xml
#Request from Francesca 12/02/2025
# Allow AI search and agent use
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: FirecrawlAgent
User-agent: AndiBot
User-agent: ExaBot
User-agent: PhindBot
User-agent: YouBot
Allow: /
# Portal6 version on September 6th
User-agent: *
User-agent: Googlebot
Disallow: /seguridad/registro
Disallow: /*listado*
Disallow: /registro
Disallow: /seguridad/acceder
Disallow: /site_images/ocu-interactivo/contact/plantilla-autorizacion.doc
Disallow: /tecnologia/nc/noticias/google-contra-consumidores*
Disallow: /LobbyAndPressAPI/*
Disallow: /errors
Disallow: /buscar
Disallow: /mi-cuenta/mi-plan-omb
#Newsletters
Disallow: /newsletters-home
Disallow: /acciones-colectivas/reclamacion-aerolineas
# Sitecore
Disallow: /sitecore
Disallow: /layouts
# Fline
Disallow: /inversiones/sitecore
Disallow: /inversiones/layouts
Disallow: /inversiones/Components
Disallow: /inversiones/cdn
Disallow: /inversiones/css
Disallow: /inversiones/ErrorPages
Disallow: /inversiones/javascript
Disallow: /inversiones/overview
Disallow: /inversiones/Properties
Disallow: /inversiones/Scripts
Disallow: /inversiones/funnel
Disallow: /fincas-y-casas/sitecore
Disallow: /fincas-y-casas/layouts
Disallow: /fincas-y-casas/Components
Disallow: /fincas-y-casas/cdn
Disallow: /fincas-y-casas/css
Disallow: /fincas-y-casas/ErrorPages
Disallow: /fincas-y-casas/javascript
Disallow: /fincas-y-casas/overview
Disallow: /fincas-y-casas/Properties
Disallow: /fincas-y-casas/Scripts
Disallow: /fincas-y-casas/funnel
#community
Disallow: /comunidad/energias-renovables/*/new-conversation
#new-aline-home
Disallow: /new-aline-home
#julio 2025
Disallow: /reclamar/lista-reclamaciones-publicas/fraude-falsedad-documental-i/78d934fe8ffcf6f906
Disallow: /reclamar/lista-reclamaciones-publicas/me-han-vetado-las-llamadas-al-/61c888cb31d80547d5
#agosto 2025
Disallow: /reclamar/lista-reclamaciones-publicas/reclamaci-C3-B3n-por-incumplimiento/af22e8392be7c0f628
Disallow: /reclamar/lista-reclamaciones-publicas/estafados-por-trabajador-empre/1f90fadd086fa8b776
Disallow: /reclamar/lista-reclamaciones-publicas/estafa/2b22540b98919b58d0
Disallow: /reclamar/lista-reclamaciones-publicas/cobro-cuota-cuenta/c23b309b7a8cee3bd1
Disallow: /reclamar/lista-reclamaciones-publicas/reembolso-estaf-/2b767dbb47774d08ad
Disallow: /reclamar/lista-reclamaciones-publicas/estancia-en-cambridge/a2d6096ebb0567b51a
Disallow: /reclamar/lista-reclamaciones-publicas?page=1318&status=§ors=
Disallow: /reclamar/lista-reclamaciones-publicas?page=430
Disallow: /reclamar/lista-reclamaciones-publicas?page=249&issue=a24MI000000TIg4YAG
Disallow: /reclamar/lista-reclamaciones-publicas/mala-inmersi-C3-B3n-ling-C3-BC-C3-ADstica-en-/21a26edfb8eff5fda7
Disallow: /reclamar/lista-reclamaciones-publicas/estancia-en-cambridge/a2d6096ebb0567b51a
Disallow: /reclamar/lista-reclamaciones-publicas/denegaci-C3-B3n-del-derecho-de-desi/50544ef8ebd4dad722
Disallow: /reclamar/lista-reclamaciones-publicas/no-me-ha-llegado-aun-mi-paquet/a37460011d51e2d61b
Disallow: /reclamar/lista-reclamaciones-publicas/problema-con-cambio-fecha-vuel/49a0beaef8fe5be82e
Disallow: /reclamar/lista-reclamaciones-publicas/reparaci-C3-B3n-placas-estropeadas-/2b51153cdc1cca07e6
Disallow: /reclamar/lista-reclamaciones-publicas/dar-de-baja-sistema-de-cobros/b4aa16e8878dabeb92
Disallow: /reclamar/lista-reclamaciones-publicas/estimaci-C3-B3n-factura-gas/9e29637c991537f182
Disallow: /reclamar/lista-reclamaciones-publicas/no-recibo-facturas-desde-julio/CPTES02010568-49
Disallow: /reclamar/lista-reclamaciones-publicas/no-llega-el-reembolso/b59c74a8350417951e
Disallow: /reclamar/lista-reclamaciones-publicas/supresi-C3-B3n-de-mi-datos-y-archiv/CPTES02002150-70
Disallow: /reclamar/lista-reclamaciones-publicas/paquete-consta-entregado-y-no-/f5d30ccf068ab90050
Disallow: /reclamar/lista-reclamaciones-publicas/problema-con-el-reembolso/1337a49c5b4da8130b
Disallow: /reclamar/lista-reclamaciones-publicas/cobro-no-autorizado-de-seguro/d97d1d43731000589f
Disallow: /reclamar/lista-reclamaciones-publicas/alerta-suplantacion-datos-pers/753d0bfb140bc08814
Disallow: /reclamar/lista-reclamaciones-publicas/no-devuelven-dinero-que-no-est/af14724734d1b42f7a
Disallow: /reclamar/lista-reclamaciones-publicas/disputa-rechazada/606634f9c52c74dec4
Disallow: /reclamar/lista-reclamaciones-publicas/no-me-dejan-devolver-el-produc/1fba6dfe7b9f7d8128
Disallow: /reclamar/lista-reclamaciones-publicas/prometen-una-transferencia-que/87a62dc6216f3a11cd
Disallow: /reclamar/lista-reclamaciones-publicas/baja-de-los-contratos/6c79cb2e3dee59c563
Disallow: /reclamar/lista-reclamaciones-publicas/reembolso-incompleto/8f3c2cac7687124638
Disallow: /reclamar/lista-reclamaciones-publicas/darme-de-baja-en-el-contrato/fd23343981f6688c4d
Disallow: /reclamar/lista-reclamaciones-publicas/problema-con-cargo-por-suscrip/197bb7f5f5c3860301
Disallow: /reclamar/lista-reclamaciones-publicas/problema-con-reembolso/eed01e79948076a2b4
Disallow: /reclamar/lista-reclamaciones-publicas/problema-con-el-reembolso/40ad297636f6916b9a
Disallow: /reclamar/lista-reclamaciones-publicas/problema-con-el-reembolso/fc3e4bb56724088864
Disallow: /reclamar/lista-reclamaciones-publicas/cuenta-bloqueada-desde-diciemb/8190034562acdaa37a
Disallow: /reclamar/lista-reclamaciones-publicas/recogida-de-una-devolucion/CPTES01949719-19
Disallow: /reclamar/lista-reclamaciones-publicas/problema-con-la-recogida-de-un/63c04fa19b37e2776c
Disallow: /reclamar/lista-reclamaciones-publicas/perdida-de-paquete/ebdfa06be0fed5ee77
Disallow: /reclamar/lista-reclamaciones-publicas/problemas-con-la-perdida-de-pa/421520b58b2157e236
Disallo- llms.txt
- absent
- ai.txt
- absent
- Sitemap
- /sitemap.xml (8 URLs)
- Feeds
- https://feeds.feedburner.com/ocu/kC8ETss6bt0
- Schema types
- Organization, ContactPoint, PostalAddress, ImageObject, QuantitativeValue, SearchAction, PropertyValueSpecification, EntryPoint
- Edge
- not identified
- Final URL
- https://www.ocu.org/
- HTML size
- 294 KB, 1,983 words of text
Compare it with a competitor head to head
Put ocu.org next to another domain and see which one an answer engine can actually read. Both sides are measured the same way, so the difference is the finding.
Already measured: vs openstack.org · vs fee.org · vs osgeo.org
Publish the score free
Sites that score well embed the badge. It links back to this live report, which re-measures on every scan.
<a href="https://crawlcensus.com/site/ocu.org"><img src="https://crawlcensus.com/badge/ocu.org.svg" alt="AI access score for ocu.org" width="150" height="20"></a>Track changes on this domain free for one
Crawler policy is edited quietly. We re-scan monitored domains daily, keep the history, and email you the moment a crawler is blocked or unblocked, an llms.txt appears, or the score moves.
Free for one domain per address, and always free for domains you have verified you own. No newsletter, only change alerts.
Pro follows 25 domains, keeps 90 days of history, compares them against each other and exports the lot as CSV, for 19 dollars a month. That is the difference: measuring is free, watching a portfolio is not.