AI access report

planetadelibros.com

Open to answer engines, blocks 4 training crawlers.

4 failing 3 partial 19 passing Scanned 23 hours ago · 2 scans on record

Reach

38.6 / 40

Whether AI crawlers and answer-engine fetchers are permitted to request the page at all. What happens on the wire outweighs what the policy file says: a live refusal costs 16 of these 40 points, twice what the heaviest robots.txt rule costs, because a page a crawler cannot fetch is unreadable no matter how well it is marked up.

Readability

21.6 / 25

Whether a fetcher that does not execute JavaScript receives the actual content, in markup an extractor can segment.

Structure

16 / 20

Machine-readable markup that states the page's type, entities, canonical URL, and discrete facts instead of leaving them to be inferred.

Attribution

4 / 12

Signals that let an answer engine name the author, date the content, resolve the publisher, and cite it under known terms.

Who is allowed to read this site 0 of 24 answer engines blocked

Crawler	Operator	Uses content for	robots.txt	Live request
GPTBot Crawls content that may be used to train OpenAI's generative AI foundation models.	OpenAI	Model training	allowed	served 200
OAI-SearchBot Indexes pages so they can be surfaced and cited in ChatGPT search results, not for training.	OpenAI	Answer index	allowed	served 200
ChatGPT-User Fetches a page when a ChatGPT user or GPT Action asks for it; user-initiated, so robots rules may not apply.	OpenAI	Live retrieval	allowed	not probed
OAI-AdsBot Visits pages submitted as ChatGPT ads to check policy compliance and ad relevance; not used for model training.	OpenAI	Live retrieval	allowed	not probed
ClaudeBot Collects web content that may contribute to training Anthropic's models; honors Crawl-delay.	Anthropic	Model training	allowed	served 200
Claude-User Retrieves pages on demand when a Claude user's question needs live web content.	Anthropic	Live retrieval	allowed	not probed
Claude-SearchBot Indexes content to improve the relevance and accuracy of Claude's search results.	Anthropic	Answer index	allowed	not probed
anthropic-ai Legacy token widely blocked for Anthropic training; Anthropic now documents ClaudeBot, Claude-User and Claude-SearchBot.	Anthropic	Model training	allowed	not probed
Google-Extended Control token with no user agent of its own; governs Gemini training and grounding use of Googlebot data.	Google	Model training	allowed	not probed
Googlebot Crawls and renders pages for Google Search, Images, Video, News and Discover.	Google	Answer index	allowed	not probed
Googlebot-News Robots token controlling Google News inclusion; crawling itself uses the Googlebot user agents.	Google	Answer index	allowed	not probed
Google-CloudVertexBot Crawls sites at a site owner's request to build Vertex AI agents; no effect on Google Search.	Google	Live retrieval	allowed	not probed
GoogleOther Generic Google crawler used by product teams for one-off fetches such as internal research and development.	Google	Model training	allowed	not probed
Applebot Crawls for Siri, Spotlight and Safari search; falls back to Googlebot rules and ignores Crawl-delay.	Apple	Answer index	allowed	not probed
Applebot-Extended Control token with no user agent; disallowing it excludes crawled content from Apple foundation model training.	Apple	Model training	allowed	not probed
Bingbot Indexes pages for Bing search and the Copilot answers that are grounded in the Bing index.	Microsoft	Answer index	allowed	not probed
msnbot Legacy Microsoft search crawler token still honored alongside bingbot.	Microsoft	Answer index	allowed	not probed
PerplexityBot Indexes and links pages in Perplexity search results; not used to collect foundation model training data.	Perplexity	Answer index	allowed	served 200
Perplexity-User ignores robots Fetches a page for a specific user question; Perplexity documents that it generally ignores robots.txt.	Perplexity	Live retrieval	allowed	not probed
Meta-ExternalAgent Crawls the web to train Meta's foundation AI models and to index content directly into products.	Meta	Model training	blocked Disallow: /	not probed
Meta-ExternalFetcher ignores robots Fetches individual links for agentic AI tasks; Meta documents that it may bypass robots.txt.	Meta	Live retrieval	allowed	not probed
FacebookBot Crawls public pages to improve language models behind Meta's speech recognition technology.	Meta	Model training	allowed	not probed
Meta-WebIndexer Indexes pages so Meta AI can cite and link them in its search answers.	Meta	Answer index	allowed	not probed
Meta-ExternalAds Crawls the web to improve Meta's advertising and other business products and services.	Meta	Model training	allowed	not probed
facebookexternalhit ignores robots Fetches shared links for Facebook, Instagram and Messenger previews; may bypass robots.txt for integrity checks.	Meta	Live retrieval	allowed	not probed
Bytespider ignores robots Downloads content to train ByteDance LLMs and is widely reported to ignore robots.txt directives.	ByteDance	Model training	blocked Disallow: /	not probed
TikTokSpider ignores robots Fetches shared URLs for TikTok link previews and feeds; not expected to follow robots.txt.	ByteDance	Live retrieval	allowed	not probed
Amazonbot Crawls for Amazon product and Alexa answers and may use the content to train Amazon AI models.	Amazon	Model training	blocked Disallow: /	not probed
Amzn-SearchBot Indexes content for Amazon search experiences such as Alexa; does not crawl for generative AI training.	Amazon	Answer index	allowed	not probed
Amzn-User ignores robots Fetches live pages to answer a user's Alexa question; Amazon documents it may not follow all robots.txt rules.	Amazon	Live retrieval	allowed	not probed
CCBot Builds the open Common Crawl web archive, a common source of LLM pretraining corpora.	Common Crawl Foundation	Archive	allowed	not probed
Diffbot Extracts structured page data for Diffbot's knowledge graph, which is licensed to AI customers.	Diffbot	Model training	allowed	not probed
omgili Collects forum, news and blog content that Webz.io sells as web data feeds, including for AI training.	Webz.io	Model training	allowed	not probed
omgilibot Legacy Omgili search crawler token still blocked alongside the current omgili agent.	Webz.io	Model training	allowed	not probed
AI2Bot Collects web text for Ai2's open datasets used to train open language models such as OLMo.	Allen Institute for AI	Model training	allowed	not probed
cohere-ai Retrieves pages to answer user-initiated prompts in Cohere's enterprise AI products.	Cohere	Live retrieval	allowed	not probed
cohere-training-data-crawler Downloads training data for the large language models behind Cohere's enterprise AI products.	Cohere	Model training	allowed	not probed
MistralAI-User Fetches pages on demand so Mistral's Vibe assistant can answer a question with live, cited web content.	Mistral AI	Live retrieval	allowed	not probed
MistralAI-Index Indexes content for Mistral search behind Vibe answers; not used for generative AI training.	Mistral AI	Answer index	allowed	not probed
MistralAI-Training Crawls web content to build datasets for training Mistral's generative AI models.	Mistral AI	Model training	allowed	not probed
DuckAssistBot Crawls pages in real time for DuckDuckGo's cited AI-assisted answers; not used for model training.	DuckDuckGo	Live retrieval	allowed	not probed
YouBot Indexes pages for You.com search results and the AI answers built on that index.	You.com	Answer index	allowed	not probed
PanguBot Collects web content used to train Huawei's PanGu family of large models.	Huawei	Model training	allowed	not probed
Timpibot Crawls pages for Timpi's decentralized index, which is also used as LLM training data.	Timpi	Model training	allowed	not probed
ImagesiftBot Downloads public images plus surrounding text to build ImageSift's searchable image index.	ImageSift (Hive)	Model training	allowed	not probed
Kangaroo Bot Scrapes site content into datasets used to train the Kangaroo LLM.	Kangaroo LLM	Model training	allowed	not probed
SemrushBot-OCOB Crawls pages to feed Semrush's ContentShake AI writing tool.	Semrush	Model training	allowed	not probed
Scrapy Generic scraping framework often used to build AI training datasets; obeys robots.txt only when ROBOTSTXT_OBEY is on.	Zyte (open-source framework)	Model training	blocked Disallow: /	not probed

Two of these columns matter differently. robots.txt is what the site declares. Live request is what actually happened when we sent a real request using that crawler's user agent from a datacentre IP, which is how edge blocking, rate limits and challenge pages show up even when robots.txt looks permissive.

Reach 38.6 / 40

Training crawlers are disallowed for this path

Blocked by robots.txt: Meta-ExternalAgent, Bytespider, Amazonbot, Scrapy.

Why it matters. GPTBot, ClaudeBot, and CCBot collect pages into pretraining corpora, and Google-Extended and Applebot-Extended are opt-out tokens that govern whether already-crawled pages may be used for Gemini and Apple Intelligence. Disallowing them removes your text from the corpora models generalise from, which is a policy choice, not a bug.
Fix. Decide this deliberately rather than by inheriting a template. If you want your content in model weights, remove the `Disallow: /` groups for `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, and `Applebot-Extended`. If you do not, keep those groups but scope them to the paths that matter and leave retrieval fetchers untouched, because `Google-Extended` and `Applebot-Extended` only control training use and never affect search or answer citation. Record the decision somewhere durable so the next robots.txt edit does not silently reverse it.

# Opt out of model training only.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Reference

5 pt

robots.txt sets Crawl-delay, which most fetchers ignore

Crawl-delay of 5s is declared. Large crawlers may fetch only a few pages per hour, or ignore the directive entirely.

Why it matters. `Crawl-delay` is not in RFC 9309. Googlebot and OpenAI's fetchers ignore it, while some retrieval agents that do honour it will space requests seconds apart, which can exceed an answer engine's fetch timeout and leave the page uncited.
Fix. Remove `Crawl-delay` and handle load at the edge instead, with caching and rate limiting keyed on the client. If crawl volume is the real problem, cache HTML at your CDN so repeat fetches never reach the origin. Keep the file to standard directives (`User-agent`, `Allow`, `Disallow`, `Sitemap`) so parser behaviour is predictable.

# Remove non-standard throttling directives:
# Crawl-delay: 10

User-agent: *
Allow: /

Reference

1 pt

✓

AI user agents receive the same 200 response as browsers

Live requests as 4 AI user agents were served normally.

16 pt

✓

Answer-engine fetchers are allowed to retrieve and cite this page

All 24 answer-engine fetchers are allowed to retrieve pages for citation.

8 pt

✓

No blanket disallow applies to this path

The wildcard group does not disallow the entire site.

4 pt

✓

robots.txt served as plain text with a 200 response

robots.txt served, 14234 bytes, 5 group(s).

2 pt

✓

Page is indexable, with no noindex directive

No noindex directive on the homepage.

2 pt

✓

Full-length snippet extraction is permitted

Snippets are not restricted by meta tags.

1 pt

✓

X-Robots-Tag header is absent or permissive

No restrictive X-Robots-Tag header.

1 pt

Readability 21.6 / 25

✗

Visible text is a small fraction of the HTML payload

Text is 1.1% of the 1465 KB document; 1062 KB is inline script.

Why it matters. Extractors strip scripts, styles, and wrapper markup before passing text to a model, and a page where content is a small fraction of the payload loses more of it to boilerplate removal. Inline JSON state blobs and deeply nested wrappers also push real text past the truncation limit of a retrieval context.
Fix. Move inline hydration state and large inline scripts out of the document, or fetch them after load instead of embedding them. Flatten wrapper `div` trees and let semantic elements carry the content, and keep utility-class soup out of the article body. Serve the same text without the boilerplate at a stable URL if you need a clean extraction target.

4 pt

✓

Substantive text is present in the server-rendered HTML

2531 words of text are present in the raw HTML. Most AI fetchers do not run JavaScript.

9 pt

✓

Primary content is wrapped in a semantic landmark

A <main> landmark marks the primary content.

3 pt

✓

Headings form a single, ordered outline

1 H1 and 15 headings total.

3 pt

✓

Title is unique and describes the page in specific terms

Title is 47 characters: "PlanetadeLibros | Novelas, ebooks y audiolibros"

3 pt

✓

Meta description provides an author-written summary

Meta description is 152 characters.

2 pt

✓

Document language is declared on the html element

Declared language: es.

1 pt

Structure 16 / 20

✗

Structured data is too generic for what the page is about

No entity types that answer engines consume.

Why it matters. Answer engines route by entity type: a `Product` node supplies price and availability, an `Article` node supplies author and dates, and a `FAQPage` node supplies question and answer pairs. A generic `WebPage` or `WebSite` node on a product or article page carries none of those fields, so the specific facts stay unavailable.
Fix. Replace bare `WebPage` and `WebSite` nodes with the most specific type that describes the page, and fill the properties that type defines. Use `@graph` to publish several linked nodes on one page, such as an `Article` whose `publisher` points at an `Organization` node by `@id`. Add `BreadcrumbList` for hierarchy and reuse the same `@id` values across pages so the entity resolves to one record.

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Product",
      "@id": "https://example.com/products/widget#product",
      "name": "Widget Pro",
      "sku": "WGT-PRO-1",
      "brand": { "@type": "Brand", "name": "Example" },
      "offers": {
        "@type": "Offer",
        "url": "https://example.com/products/widget",
        "price": "49.00",
        "priceCurrency": "USD",
        "availability": "https://schema.org/InStock"
      }
    },
    {
      "@type": "BreadcrumbList",
      "itemListElement": [
        { "@type": "ListItem", "position": 1, "name": "Products", "item": "https://example.com/products" },
        { "@type": "ListItem", "position": 2, "name": "Widget Pro" }
      ]
    }
  ]
}

Reference

4 pt

✓

Page ships JSON-LD structured data

155 JSON-LD node(s): ItemList, ListItem, Book, Person.

7 pt

✓

JSON-LD parses cleanly with recognised schema.org terms

All JSON-LD blocks parse cleanly.

3 pt

✓

Page declares a self-referential canonical URL

Canonical: https://www.planetadelibros.com

3 pt

✓

XML sitemap is declared in robots.txt and returns 200

Sitemap found at /sitemap.xml (16 URLs on the first document).

2 pt

✓

Key facts are available in lists or tables

0 tables, 72 lists, 0 code blocks, 0 question headings.

1 pt

Attribution 4 / 12

✗

No /llms.txt index of canonical pages

No /llms.txt.

Why it matters. `/llms.txt` is a markdown file that points an assistant at the canonical pages for a site, so retrieval does not depend on which page a search happened to return. Its format is fixed: one `#` title, a `>` blockquote summary, then `##` sections of markdown links with short notes.
Fix. Publish `/llms.txt` as `text/plain` markdown: an `#` H1 with the project name, a `>` blockquote summary, optional plain paragraphs of context, then `##` sections whose bullets are `[title](absolute-url): note`. Link the pages you want quoted, put lower-priority links under an `## Optional` section, and prefer URLs that also serve clean markdown. Keep it generated from the same source as your sitemap so it does not drift, and remember it is a hint for assistants, not an access control mechanism.

# Example

> Example publishes reference documentation for the Widget API and guides for
> configuring crawler access.

Prefer the pages below over search results; each URL is canonical.

## Docs

- [Widget API reference](https://example.com/docs/api): endpoints, auth, limits.
- [Quickstart](https://example.com/docs/quickstart): first request in five minutes.

## Policies

- [Crawler policy](https://example.com/legal/crawlers): which agents we allow.

## Optional

- [Changelog](https://example.com/changelog): dated release notes.

Reference

4 pt

✗

No Organization entity identifies the publisher

No Organization entity, so the brand is harder to resolve to a known entity.

Why it matters. An `Organization` node with `sameAs` links resolves your site to a single entity across knowledge graphs instead of leaving the publisher name as an ambiguous string. That resolution is what lets an assistant attribute an answer to your brand and reuse your name, logo, and contact details.
Fix. Publish one `Organization` node, usually on the home page, with `name`, `url`, `logo`, and `sameAs` pointing at the profiles that already describe you: Wikipedia or Wikidata, Crunchbase, LinkedIn, and your primary social accounts. Give it a stable `@id` such as `https://example.com/#organization` and reference that `@id` from each page's `publisher` property instead of repeating the block. Add `contactPoint` and `address` when they are public, and keep every value identical to what the site shows.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "@id": "https://example.com/#organization",
  "name": "Example",
  "legalName": "Example Holdings, Inc.",
  "url": "https://example.com",
  "logo": {
    "@type": "ImageObject",
    "url": "https://example.com/logo.png",
    "width": 512,
    "height": 512
  },
  "sameAs": [
    "https://www.wikidata.org/wiki/Q00000000",
    "https://www.linkedin.com/company/example",
    "https://github.com/example"
  ],
  "contactPoint": {
    "@type": "ContactPoint",
    "contactType": "customer support",
    "email": "support@example.com",
    "areaServed": "US",
    "availableLanguage": ["en"]
  }
}

Reference

3 pt

No machine-readable license or usage terms for the content

No licence declaration, so reuse terms are ambiguous.

Why it matters. A `license` property, or a linked terms page, states the reuse conditions in a place a crawler can read, rather than leaving them to be inferred. Where terms are unstated, some pipelines default to the more restrictive handling, which reduces how much of the text is quoted.
Fix. Add a `license` property to the page's structured data pointing at a specific license URL, such as a Creative Commons deed or your own terms page, and add `rel="license"` on the visible link. Use `usageInfo` for conditions that are not a standard license, such as attribution wording or an API-only clause. State the terms once, at a stable URL, and reference it from every page rather than restating it per template.

<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Allow AI answer engines in robots.txt",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "usageInfo": "https://example.com/legal/content-reuse",
  "creditText": "Example, crawlcensus research desk"
}
</script>

Reference

2 pt

–

No machine-readable published or modified date

This page declares no article-type structured data, so publication and modification dates are not expected of it. The check is excluded from the score rather than counted as a failure.

Why it matters. Answer engines prefer recent sources for questions about current state and use `dateModified` to decide whether a cached copy needs refetching. With no machine-readable date the page is treated as undated and loses to competitors that publish one.
Fix. Publish `datePublished` and `dateModified` in the page's structured data as ISO 8601 values with a timezone offset. Update `dateModified` only when the content actually changes, since bumping it on every deploy trains crawlers to ignore it. Mirror the value in a visible `<time datetime>` element so the rendered text and the metadata agree, and keep the sitemap `lastmod` consistent with it.

<time datetime="2026-08-04T14:20:00-04:00">Updated August 4, 2026</time>

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Allow AI answer engines in robots.txt",
  "datePublished": "2026-02-11T09:00:00-05:00",
  "dateModified": "2026-08-04T14:20:00-04:00"
}
</script>

Reference

3 pt

✓

Author is declared in structured data

Authorship is declared in structured data or markup.

3 pt

Evidence raw measurements

robots.txt

Size: 14,234 bytes
Groups: 5
Sitemaps: https://www.planetadelibros.com/sitemap.xml

View the file as our crawler received it

# Robots V1.7

User-agent: *

#Bloqueo del crawling de ciertos directorios y del buscador
Disallow: /usuaris/
Allow: /usuaris/*.webp
Disallow: /*?textoBuscador=
Disallow: /*&textoBuscador=
Disallow: /*?access_token=*
Disallow: /preview/pagina/
Disallow: /usuaris/web_plataformas_venda/fotos/
Disallow: /contacto/*.txt
Disallow: /contacte/*.txt

#Permitiendo recursos del directorio usuaris
Allow: /usuaris/*/fotos/
Allow: /usuaris/*.pdf

# Permitiendo recursos de JS y CSS
Allow: *.js
Allow: *.css
Allow: *.json
Disallow: /js/obfuscator.js

# Permitiendo páginas importantes
Allow: /usuaris/libros_contenido/arxius/1/32_1_Lalineadeltiempodelanovela.doc

#Sitemap
Sitemap: https://www.planetadelibros.com/sitemap.xml

User-agent: Twitterbot
Allow: /

User-agent: BLEXBot
User-agent: Cliqzbot
User-agent: MojeekBot
User-agent: musobot
User-agent: linkdexbot
User-agent: Seznambot
User-agent: MJ12bot
User-agent: EveryoneSocialBot
User-agent: GnowitNewsbot
User-agent: Leikibot
User-agent: OutclicksBot
User-agent: sudorlybot
User-agent: woobot
User-agent: linkdexbot
User-agent: epicbot
User-agent: Tweetmemebot
User-agent: obot
User-agent: yoozbot
User-agent: MSN-bot
Disallow: /

# Users inutiles
User-agent: 360Spider
User-agent: 404checker
User-agent: 404enemy
User-agent: 80legs
User-agent: Abonti
User-agent: Aboundex
User-agent: Acunetix
User-agent: Adidum/1.0
User-agent: ADmantX
User-agent: AfD-Verbotsverfahren
User-agent: AIBOT
User-agent: AiHitBot
User-agent: Aipbot
User-agent: Alexibot
User-agent: Alligator
User-agent: AllSubmitter
User-agent: AlphaBot
User-agent: Anarchie
User-agent: Apexoo
User-agent: archive.org_bot
User-agent: ASPSeek
User-agent: Asterias
User-agent: Attach
User-agent: autoemailspider
User-agent: BackDoorBot
User-agent: Backlink-Ceck
User-agent: backlink-check
User-agent: BacklinkCrawler
User-agent: BackStreet
User-agent: BackWeb
User-agent: Badass
User-agent: Bandit
User-agent: Barkrowler
User-agent: BatchFTP
User-agent: Battleztar Bazinga
User-agent: BBBike
User-agent: BDCbot
User-agent: BDFetch
User-agent: BetaBot
User-agent: Bigfoot
User-agent: Bitacle
User-agent: Black Hole
User-agent: Blackboard
User-agent: BlackWidow
User-agent: BLEXBot/1.0
User-agent: Blow
User-agent: BlowFish
User-agent: Boardreader
User-agent: Bolt
User-agent: BotALot
User-agent: Brandprotect
User-agent: Brandwatch
User-agent: BREW-Applet/0x20068888
User-agent: Bubing
User-agent: Buddy
User-agent: BuiltBotTough
User-agent: BuiltWith
User-agent: Bullseye
User-agent: BunnySlippers
User-agent: BuzzSumo
User-agent: Calculon
User-agent: CATExplorador
User-agent: CazoodleBot
User-agent: Cegbfeieh
User-agent: CFNetwork/760.2.6
User-agent: CheeseBot
User-agent: CherryPicker
User-agent: CheTeam
User-agent: ChinaClaw
User-agent: Chlooe
User-agent: Claritybot
User-agent: Cloud mapping
User-agent: coccoc/1.0
User-agent: coccocbot-web
User-agent: Cogentbot
User-agent: cognitiveseo
User-agent: Collector
User-agent: com.plumanalytics
User-agent: Copier
User-agent: CopyRightCheck
User-agent: Copyscape
User-agent: Cosmos
User-agent: Craftbot
User-agent: crawl.sogou.com
User-agent: crawler.feedback
User-agent: crawler4j
User-agent: CrazyWebCrawler
User-agent: Crescent
User-agent: CSHttp
User-agent: Curious
User-agent: cURL
User-agent: Custo
User-agent: DatabaseDriverMysqli
User-agent: DataCha0s
User-agent: DBLBot
User-agent: demandbase-bot
User-agent: Demon
User-agent: Deusu
User-agent: Devil
User-agent: Digincore
User-agent: DigitalPebble
User-agent: DIIbot
User-agent: Dirbuster
User-agent: Disco
User-agent: Discobot
User-agent: Discoverybot
User-agent: DittoSpyder
User-agent: DnyzBot
User-agent: DOC
User-agent: DomainAppender
User-agent: DomainCrawler
User-agent: DomainSigmaCrawler
User-agent: DomainStatsBot
User-agent: Dotbot
User-agent: DotBot/1.1
User-agent: Download Ninja
User-agent: Download Wonder
User-agent: Dragonfly
User-agent: Drip
User-agent: DTS Agent
User-agent: EasyDL
User-agent: Ebingbong
User-agent: eCatch
User-agent: ECCP/1.0
User-agent: Ecxi
User-agent: EirGrabber
User-agent: EMail Siphon
User-agent: EMail Wolf
User-agent: EroCrawler
User-agent: evc-batch
User-agent: Evil
User-agent: Exabot
User-agent: Exabot/3.0
User-agent: Exalead Previewer
User-agent: Express WebPictures
User-agent: ExtLinksBot
User-agent: Extractor
User-agent: ExtractorPro
User-agent: Extreme Picture Finder
User-agent: EyeNetIE
User-agent: Ezooms
User-agent: fast
User-agent: FDM
User-agent: FemtosearchBot
User-agent: Fetch
User-agent: FHscan
User-agent: Fimap
User-agent: FlashGet
User-agent: Flunky
User-agent: Foobot
User-agent: Freeuploader
User-agent: FrontPage
User-agent: FyberSpider
User-agent: Fyrebot
User-agent: G-i-g-a-b-o-t
User-agent: GalaxyBot
User-agent: Genieo
User-agent: Genieo/1.0
User-agent: GermCrawler
User-agent: Getintent
User-agent: GetRight
User-agent: GetWeb
User-agent: Gigablast
User-agent: Gigabot
User-agent: Gigabot/1.0
User-agent: Go 1.1 package
User-agent: Go!Zilla
User-agent: Go-Ahead-Got-It
User-agent: Gotit
User-agent: GoZilla
User-agent: Grabber
User-agent: GrabNet
User-agent: Grafula
User-agent: GrapeFX
User-agent: GrapeshotCrawler
User-agent: GrapeshotCrawler/2.0
User-agent: GridBot
User-agent: grub-client
User-agent: GT::WWW
User-agent: Haansoft
User-agent: HaosouSpider
User-agent: Harvest
User-agent: Havij
User-agent: HEADMasterSEO
User-agent: Heritrix
User-agent: heritrix/1.12.1
User-agent: Hloader
User-agent: HMView
User-agent: HTMLparser
User-agent: HTTP::Lite
User-agent: HTTrack
User-agent: HTTrack 3.0x
User-agent: Humanlinks
User-agent: HybridBot
User-agent: Iblog
User-agent: Id-search
User-agent: IDBot
User-agent: IlseBot
User-agent: Image Fetch
User-agent: Image Sucker
User-agent: IndeedBot
User-agent: Indy Library
User-agent: InfoNaviRobot
User-agent: InfoTekies
User-agent: instabid
User-agent: Intelliseek
User-agent: InterGET
User-agent: Internet Ninja
User-agent: InternetSeer
User-agent: internetVista monitor
User-agent: ips-agent
User-agent: Iria
User-agent: IRLbot
User-agent: Iskanie
U

Machine-readable extras

llms.txt: absent
ai.txt: absent
Sitemap: /sitemap.xml (16 URLs)
Feeds: none
Schema types: ItemList, ListItem, Book, Person
Edge: not identified
Final URL: https://www.planetadelibros.com/
HTML size: 1,465 KB, 2,531 words of text

Compare it with a competitor head to head

Put planetadelibros.com next to another domain and see which one an answer engine can actually read. Both sides are measured the same way, so the difference is the finding.

Publish the score free

Sites that score well embed the badge. It links back to this live report, which re-measures on every scan.

<a href="https://crawlcensus.com/site/planetadelibros.com"><img src="https://crawlcensus.com/badge/planetadelibros.com.svg" alt="AI access score for planetadelibros.com" width="150" height="20"></a>

Track changes on this domain free for one

Crawler policy is edited quietly. We re-scan monitored domains daily, keep the history, and email you the moment a crawler is blocked or unblocked, an llms.txt appears, or the score moves.

Free for one domain per address, and always free for domains you have verified you own. No newsletter, only change alerts.

Pro follows 25 domains, keeps 90 days of history, compares them against each other and exports the lot as CSV, for 19 dollars a month. That is the difference: measuring is free, watching a portfolio is not.

Reading this as a crawler? check before you fetch

Most of the traffic to these report pages is automated, so this is addressed to it. The same measurement behind this page will tell you, before you spend a request, whether a domain will actually serve your agent, whether its robots.txt permits you while the edge refuses you anyway, or whether it answers HTTP 402 because it will serve you for money.

curl -X POST https://crawlcensus.com/api/v1/preflight \
  -H 'content-type: application/json' \
  -d '{"agent":"gptbot","domains":["planetadelibros.com"]}'

Works for any crawler token, not only the ones tracked here. Also an MCP server at https://crawlcensus.com/mcp, a plain deny-list file per agent, and clients for JavaScript and Python. Everything for crawler operators.

If this census publishes something wrong about your crawler, the correction channel is on your operator page.