AI access report

bibliocommons.com

Blocks 2 of 24 answer engines, blocks 13 training crawlers.

5 failing 5 partial 17 passing Scanned 1 hour ago · 2 scans on record trend

Reach

34.5 / 40

Whether AI crawlers and answer-engine fetchers are permitted to request the page at all, in robots.txt, in robots directives, and at the edge.

Readability

22.5 / 25

Whether a fetcher that does not execute JavaScript receives the actual content, in markup an extractor can segment.

Structure

16.5 / 20

Machine-readable markup that states the page's type, entities, canonical URL, and discrete facts instead of leaving them to be inferred.

Attribution

4 / 15

Signals that let an answer engine name the author, date the content, resolve the publisher, and cite it under known terms.

Who is allowed to read this site 2 of 24 answer engines blocked

Crawler	Operator	Uses content for	robots.txt	Live request
GPTBot Crawls content that may be used to train OpenAI's generative AI foundation models.	OpenAI	Model training	blocked Disallow: /	served 200
OAI-SearchBot Indexes pages so they can be surfaced and cited in ChatGPT search results, not for training.	OpenAI	Answer index	allowed Allow: /	served 200
ChatGPT-User Fetches a page when a ChatGPT user or GPT Action asks for it; user-initiated, so robots rules may not apply.	OpenAI	Live retrieval	allowed Allow: /	not probed
OAI-AdsBot Visits pages submitted as ChatGPT ads to check policy compliance and ad relevance; not used for model training.	OpenAI	Live retrieval	allowed Allow: /	not probed
ClaudeBot Collects web content that may contribute to training Anthropic's models; honors Crawl-delay.	Anthropic	Model training	blocked Disallow: /	served 200
Claude-User Retrieves pages on demand when a Claude user's question needs live web content.	Anthropic	Live retrieval	allowed Allow: /	not probed
Claude-SearchBot Indexes content to improve the relevance and accuracy of Claude's search results.	Anthropic	Answer index	allowed Allow: /	not probed
anthropic-ai Legacy token widely blocked for Anthropic training; Anthropic now documents ClaudeBot, Claude-User and Claude-SearchBot.	Anthropic	Model training	blocked Disallow: /	not probed
Google-Extended Control token with no user agent of its own; governs Gemini training and grounding use of Googlebot data.	Google	Model training	blocked Disallow: /	not probed
Googlebot Crawls and renders pages for Google Search, Images, Video, News and Discover.	Google	Answer index	allowed Allow: /	not probed
Googlebot-News Robots token controlling Google News inclusion; crawling itself uses the Googlebot user agents.	Google	Answer index	allowed Allow: /	not probed
Google-CloudVertexBot Crawls sites at a site owner's request to build Vertex AI agents; no effect on Google Search.	Google	Live retrieval	allowed Allow: /	not probed
GoogleOther Generic Google crawler used by product teams for one-off fetches such as internal research and development.	Google	Model training	allowed Allow: /	not probed
Applebot Crawls for Siri, Spotlight and Safari search; falls back to Googlebot rules and ignores Crawl-delay.	Apple	Answer index	allowed Allow: /	not probed
Applebot-Extended Control token with no user agent; disallowing it excludes crawled content from Apple foundation model training.	Apple	Model training	blocked Disallow: /	not probed
Bingbot Indexes pages for Bing search and the Copilot answers that are grounded in the Bing index.	Microsoft	Answer index	allowed Allow: /	not probed
msnbot Legacy Microsoft search crawler token still honored alongside bingbot.	Microsoft	Answer index	allowed Allow: /	not probed
PerplexityBot Indexes and links pages in Perplexity search results; not used to collect foundation model training data.	Perplexity	Answer index	allowed Allow: /	served 200
Perplexity-User ignores robots Fetches a page for a specific user question; Perplexity documents that it generally ignores robots.txt.	Perplexity	Live retrieval	allowed Allow: /	not probed
Meta-ExternalAgent Crawls the web to train Meta's foundation AI models and to index content directly into products.	Meta	Model training	blocked Disallow: /	not probed
Meta-ExternalFetcher ignores robots Fetches individual links for agentic AI tasks; Meta documents that it may bypass robots.txt.	Meta	Live retrieval	allowed Allow: /	not probed
FacebookBot Crawls public pages to improve language models behind Meta's speech recognition technology.	Meta	Model training	blocked Disallow: /	not probed
Meta-WebIndexer Indexes pages so Meta AI can cite and link them in its search answers.	Meta	Answer index	allowed Allow: /	not probed
Meta-ExternalAds Crawls the web to improve Meta's advertising and other business products and services.	Meta	Model training	allowed Allow: /	not probed
facebookexternalhit ignores robots Fetches shared links for Facebook, Instagram and Messenger previews; may bypass robots.txt for integrity checks.	Meta	Live retrieval	allowed Allow: /	not probed
Bytespider ignores robots Downloads content to train ByteDance LLMs and is widely reported to ignore robots.txt directives.	ByteDance	Model training	blocked Disallow: /	not probed
TikTokSpider ignores robots Fetches shared URLs for TikTok link previews and feeds; not expected to follow robots.txt.	ByteDance	Live retrieval	allowed Allow: /	not probed
Amazonbot Crawls for Amazon product and Alexa answers and may use the content to train Amazon AI models.	Amazon	Model training	allowed Allow: /	not probed
Amzn-SearchBot Indexes content for Amazon search experiences such as Alexa; does not crawl for generative AI training.	Amazon	Answer index	allowed Allow: /	not probed
Amzn-User ignores robots Fetches live pages to answer a user's Alexa question; Amazon documents it may not follow all robots.txt rules.	Amazon	Live retrieval	allowed Allow: /	not probed
CCBot Builds the open Common Crawl web archive, a common source of LLM pretraining corpora.	Common Crawl Foundation	Archive	blocked Disallow: /	not probed
Diffbot Extracts structured page data for Diffbot's knowledge graph, which is licensed to AI customers.	Diffbot	Model training	blocked Disallow: /	not probed
omgili Collects forum, news and blog content that Webz.io sells as web data feeds, including for AI training.	Webz.io	Model training	allowed Allow: /	not probed
omgilibot Legacy Omgili search crawler token still blocked alongside the current omgili agent.	Webz.io	Model training	blocked Disallow: /	not probed
AI2Bot Collects web text for Ai2's open datasets used to train open language models such as OLMo.	Allen Institute for AI	Model training	blocked Disallow: /	not probed
cohere-ai Retrieves pages to answer user-initiated prompts in Cohere's enterprise AI products.	Cohere	Live retrieval	blocked Disallow: /	not probed
cohere-training-data-crawler Downloads training data for the large language models behind Cohere's enterprise AI products.	Cohere	Model training	allowed Allow: /	not probed
MistralAI-User Fetches pages on demand so Mistral's Vibe assistant can answer a question with live, cited web content.	Mistral AI	Live retrieval	blocked Disallow: /	not probed
MistralAI-Index Indexes content for Mistral search behind Vibe answers; not used for generative AI training.	Mistral AI	Answer index	allowed Allow: /	not probed
MistralAI-Training Crawls web content to build datasets for training Mistral's generative AI models.	Mistral AI	Model training	allowed Allow: /	not probed
DuckAssistBot Crawls pages in real time for DuckDuckGo's cited AI-assisted answers; not used for model training.	DuckDuckGo	Live retrieval	allowed Allow: /	not probed
YouBot Indexes pages for You.com search results and the AI answers built on that index.	You.com	Answer index	allowed Allow: /	not probed
PanguBot Collects web content used to train Huawei's PanGu family of large models.	Huawei	Model training	allowed Allow: /	not probed
Timpibot Crawls pages for Timpi's decentralized index, which is also used as LLM training data.	Timpi	Model training	blocked Disallow: /	not probed
ImagesiftBot Downloads public images plus surrounding text to build ImageSift's searchable image index.	ImageSift (Hive)	Model training	blocked Disallow: /	not probed
Kangaroo Bot Scrapes site content into datasets used to train the Kangaroo LLM.	Kangaroo LLM	Model training	allowed Allow: /	not probed
SemrushBot-OCOB Crawls pages to feed Semrush's ContentShake AI writing tool.	Semrush	Model training	allowed Allow: /	not probed
Scrapy Generic scraping framework often used to build AI training datasets; obeys robots.txt only when ROBOTSTXT_OBEY is on.	Zyte (open-source framework)	Model training	allowed Allow: /	not probed

Two of these columns matter differently. robots.txt is what the site declares. Live request is what actually happened when we sent a real request using that crawler's user agent from a datacentre IP, which is how edge blocking, rate limits and challenge pages show up even when robots.txt looks permissive.

Reach 34.5 / 40

Answer-engine fetchers are blocked, so you cannot be cited

Blocked from citing you: cohere-ai, MistralAI-User.

Why it matters. Retrieval fetchers (OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User) load a page at answer time to quote and link it. When robots.txt disallows them the assistant drops the URL from its candidate set, so the citation goes to a competitor while your training exposure stays exactly the same.
Fix. Separate the two crawler classes in robots.txt instead of blocking by vendor. Allow `OAI-SearchBot` and `ChatGPT-User` (OpenAI retrieval and user-initiated fetches), `Claude-SearchBot` and `Claude-User` (Anthropic retrieval), and `PerplexityBot` and `Perplexity-User`; keep any opt-out you want on `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, and `Applebot-Extended`. Order does not decide precedence in RFC 9309 parsers, the longest matching rule does, so make the allow rules at least as specific as the disallow rules. Blocking retrieval buys nothing on training, because the training crawlers are separate user agents with separate rules.

# Retrieval and citation fetchers: allowed.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Training and dataset crawlers: disallowed.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /checkout

Sitemap: https://example.com/sitemap.xml

Reference

12 pt

Training crawlers are disallowed for this path

Blocked by robots.txt: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Bytespider, Diffbot, omgilibot, AI2Bot, Timpibot, ImagesiftBot.

Why it matters. GPTBot, ClaudeBot, and CCBot collect pages into pretraining corpora, and Google-Extended and Applebot-Extended are opt-out tokens that govern whether already-crawled pages may be used for Gemini and Apple Intelligence. Disallowing them removes your text from the corpora models generalise from, which is a policy choice, not a bug.
Fix. Decide this deliberately rather than by inheriting a template. If you want your content in model weights, remove the `Disallow: /` groups for `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, and `Applebot-Extended`. If you do not, keep those groups but scope them to the paths that matter and leave retrieval fetchers untouched, because `Google-Extended` and `Applebot-Extended` only control training use and never affect search or answer citation. Record the decision somewhere durable so the next robots.txt edit does not silently reverse it.

# Opt out of model training only.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Reference

8 pt

✓

AI user agents receive the same 200 response as browsers

Live requests as 4 AI user agents were served normally.

6 pt

✓

No blanket disallow applies to this path

The wildcard group does not disallow the entire site.

5 pt

✓

robots.txt served as plain text with a 200 response

robots.txt served, 1949 bytes, 31 group(s).

3 pt

✓

No Crawl-delay directive constrains fetchers

No Crawl-delay directive.

2 pt

✓

Page is indexable, with no noindex directive

No noindex directive on the homepage.

2 pt

✓

Full-length snippet extraction is permitted

Snippets are not restricted by meta tags.

1 pt

✓

X-Robots-Tag header is absent or permissive

No restrictive X-Robots-Tag header.

1 pt

Readability 22.5 / 25

✗

No lang attribute declares the document language

No lang attribute on <html>.

Why it matters. `<html lang>` tells tokenizers and language filters which language the text is in. Without it a retrieval pipeline guesses, and a wrong guess drops the page from language-scoped candidate sets and degrades sentence splitting.
Fix. Set a valid BCP 47 tag on the root element, such as `lang="en"` or `lang="pt-BR"`. Mark inline passages in another language with `lang` on the containing element. If you publish translations, pair the declaration with `hreflang` alternates so each version is attributed to the right locale.

<html lang="en">
  <head>
    <link rel="alternate" hreflang="es" href="https://example.com/es/page" />
  </head>
</html>

Reference

1 pt

Title is missing, duplicated, or too generic to identify the page

Title is 67 characters: "BiblioCommons | Empowering Public Libraries with Digital Technology"

Why it matters. The `<title>` is the first text an indexer stores for the URL and is frequently reused verbatim as the link label in a generated answer. An empty, duplicated, or purely brand-name title gives the ranker no lexical signal for the page's actual topic.
Fix. Write a unique `<title>` of roughly 30 to 60 characters that leads with the page's subject and ends with the brand. Match the wording of the `h1` so the stored label and the visible heading agree. Remove template output such as "Home" or a bare site name, and keep keyword lists and separator chains out of it.

<title>Allow AI answer engines in robots.txt | Example</title>

Reference

3 pt

✓

Substantive text is present in the server-rendered HTML

1418 words of text are present in the raw HTML. Most AI fetchers do not run JavaScript.

9 pt

✓

Visible text makes up a healthy share of the HTML payload

Text is 8.8% of the 102 KB document; 3 KB is inline script.

4 pt

✓

Primary content is wrapped in a semantic landmark

A <main> landmark marks the primary content.

3 pt

✓

Headings form a single, ordered outline

1 H1 and 28 headings total, 2 phrased as questions.

3 pt

✓

Meta description provides an author-written summary

Meta description is 194 characters.

2 pt

Structure 16.5 / 20

✗

No canonical URL is declared for this page

No rel=canonical, so duplicate URLs split citation signals.

Why it matters. Tracking parameters, trailing slashes, and http and https variants make one document reachable at many URLs. Without `rel=canonical` the retrieval index can hold several near-duplicate records, splitting the signals that rank the page and making the cited link unstable.
Fix. Emit an absolute, self-referential `<link rel="canonical">` on every page, using the exact scheme, host, and path you want cited. Point parameterised and paginated variants at the canonical document, and make sure the canonical URL itself returns 200 rather than redirecting. Keep the canonical, the `og:url`, the sitemap entry, and the JSON-LD `url` identical, since conflicting values are treated as a weak hint and may be overridden.

<link rel="canonical" href="https://example.com/blog/robots-for-ai" />
<meta property="og:url" content="https://example.com/blog/robots-for-ai" />

Reference

3 pt

Facts are only in prose, with no list or table structure

0 tables, 0 lists, 0 code blocks, 2 question headings.

Why it matters. Lists, tables with header cells, and definition lists survive HTML-to-text conversion as discrete rows a model can lift into an answer verbatim. The same facts written as a paragraph, or laid out with positioned `div`s, lose their row and column relationships during extraction.
Fix. Put specifications, comparisons, pricing, and steps into real `<table>` markup with `<caption>` and `<th scope>`, or into `<ul>`, `<ol>`, and `<dl>` elements. Do not simulate tables with `div` grids, and avoid images of tables, which carry no extractable text. Keep one fact per row or list item so a chunk stays meaningful on its own.

<table>
  <caption>AI fetcher purposes</caption>
  <thead>
    <tr><th scope="col">User agent</th><th scope="col">Purpose</th></tr>
  </thead>
  <tbody>
    <tr><td>GPTBot</td><td>Model training</td></tr>
    <tr><td>OAI-SearchBot</td><td>Retrieval and citation</td></tr>
  </tbody>
</table>

Reference

1 pt

✓

Page ships JSON-LD structured data

13 JSON-LD node(s): Organization, ImageObject, Place, PostalAddress, ContactPoint, Country.

7 pt

✓

Structured data uses a specific type that matches the page

Recognized types: organization, website.

4 pt

✓

JSON-LD parses cleanly with recognised schema.org terms

All JSON-LD blocks parse cleanly.

3 pt

✓

XML sitemap is declared in robots.txt and returns 200

Sitemap found at https://www.bibliocommons.com/sitemap.xml.

2 pt

Attribution 4 / 15

✗

No /llms.txt index of canonical pages

No /llms.txt.

Why it matters. `/llms.txt` is a markdown file that points an assistant at the canonical pages for a site, so retrieval does not depend on which page a search happened to return. Its format is fixed: one `#` title, a `>` blockquote summary, then `##` sections of markdown links with short notes.
Fix. Publish `/llms.txt` as `text/plain` markdown: an `#` H1 with the project name, a `>` blockquote summary, optional plain paragraphs of context, then `##` sections whose bullets are `[title](absolute-url): note`. Link the pages you want quoted, put lower-priority links under an `## Optional` section, and prefer URLs that also serve clean markdown. Keep it generated from the same source as your sitemap so it does not drift, and remember it is a hint for assistants, not an access control mechanism.

# Example

> Example publishes reference documentation for the Widget API and guides for
> configuring crawler access.

Prefer the pages below over search results; each URL is canonical.

## Docs

- [Widget API reference](https://example.com/docs/api): endpoints, auth, limits.
- [Quickstart](https://example.com/docs/quickstart): first request in five minutes.

## Policies

- [Crawler policy](https://example.com/legal/crawlers): which agents we allow.

## Optional

- [Changelog](https://example.com/changelog): dated release notes.

Reference

4 pt

✗

No machine-readable author is attached to the page

No author or Person entity, which weakens the authority signals answer engines use.

Why it matters. An `author` property in structured data is what lets an answer engine name a person or organisation as the source and link the byline to a stable profile. A byline that exists only as styled text is not reliably associated with the document during extraction.
Fix. Add an `author` property to the page's `Article`, `BlogPosting`, or `NewsArticle` node, typed as `Person` or `Organization`, with a `name` and a `url` pointing at a real profile page. Give each author a stable `@id` and reuse it across posts so the entity consolidates. Keep the visible byline identical to the structured value, and avoid generic names such as "Admin" or "Staff Writer" where a real author exists.

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Allow AI answer engines in robots.txt",
  "author": {
    "@type": "Person",
    "@id": "https://example.com/authors/dana-reyes#person",
    "name": "Dana Reyes",
    "url": "https://example.com/authors/dana-reyes",
    "jobTitle": "Infrastructure Engineer",
    "sameAs": ["https://github.com/danareyes"]
  }
}

Reference

3 pt

✗

No machine-readable published or modified date

No publication or modification dates in structured data.

Why it matters. Answer engines prefer recent sources for questions about current state and use `dateModified` to decide whether a cached copy needs refetching. With no machine-readable date the page is treated as undated and loses to competitors that publish one.
Fix. Publish `datePublished` and `dateModified` in the page's structured data as ISO 8601 values with a timezone offset. Update `dateModified` only when the content actually changes, since bumping it on every deploy trains crawlers to ignore it. Mirror the value in a visible `<time datetime>` element so the rendered text and the metadata agree, and keep the sitemap `lastmod` consistent with it.

<time datetime="2026-08-04T14:20:00-04:00">Updated August 4, 2026</time>

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Allow AI answer engines in robots.txt",
  "datePublished": "2026-02-11T09:00:00-05:00",
  "dateModified": "2026-08-04T14:20:00-04:00"
}
</script>

Reference

3 pt

No machine-readable license or usage terms for the content

No licence declaration, so reuse terms are ambiguous.

Why it matters. A `license` property, or a linked terms page, states the reuse conditions in a place a crawler can read, rather than leaving them to be inferred. Where terms are unstated, some pipelines default to the more restrictive handling, which reduces how much of the text is quoted.
Fix. Add a `license` property to the page's structured data pointing at a specific license URL, such as a Creative Commons deed or your own terms page, and add `rel="license"` on the visible link. Use `usageInfo` for conditions that are not a standard license, such as attribution wording or an API-only clause. State the terms once, at a stable URL, and reference it from every page rather than restating it per template.

<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Allow AI answer engines in robots.txt",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "usageInfo": "https://example.com/legal/content-reuse",
  "creditText": "Example, crawlcensus research desk"
}
</script>

Reference

2 pt

✓

Organization entity declares the publisher and its identifiers

Organization entity with sameAs links is present.

3 pt

Evidence raw measurements

robots.txt

Size: 1,949 bytes
Groups: 31
Sitemaps: https://www.bibliocommons.com/sitemap.xml

View the file as our crawler received it

# robots.txt for www.bibliocommons.com
# Marketing site — allow standard SEO crawlers and AI answer engines.
# Block AI training crawlers (content-ownership decision; no SEO/AEO impact).

# ============================================
# Default: allow all crawlers to everything.
# Named blocks below override this for specific bots.
# ============================================
User-agent: *
Allow: /

Sitemap: https://www.bibliocommons.com/sitemap.xml

# ============================================
# AI Answer Engines / real-time browsing bots
# ALLOWED — these drive AEO/GEO citations.
# (Listed for explicit intent; they'd be allowed by the * rule anyway.)
# ============================================
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GoogleOther
Allow: /

User-agent: Applebot
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

# ============================================
# AI Training Crawlers
# DISALLOWED — content-ownership decision.
# ============================================
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: AI2Bot
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: YandexGPT
Disallow: /

User-agent: MistralAI-User
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: SBIntuitionsBot
Disallow: /

Machine-readable extras

llms.txt: absent
ai.txt: absent
Sitemap: https://www.bibliocommons.com/sitemap.xml
Feeds: none
Schema types: Organization, ImageObject, Place, PostalAddress, ContactPoint, Country, WebSite, WebPage
Edge: Cloudflare
Final URL: https://bibliocommons.com/
HTML size: 102 KB, 1,418 words of text

Publish the score free

Sites that score well embed the badge. It links back to this live report, which re-measures on every scan.

<a href="https://crawlcensus.com/site/bibliocommons.com"><img src="https://crawlcensus.com/badge/bibliocommons.com.svg" alt="AI access score for bibliocommons.com" width="150" height="20"></a>

Track changes on this domain pro

Crawler policy is edited quietly. We re-scan monitored domains daily, keep the history, and email you the moment a crawler is blocked or unblocked, an llms.txt appears, or the score moves.

One address, unlimited domains during the beta. No newsletter, only change alerts.