AI access report
esxdos.org
Open to answer engines, no structured data.
12 failing
3 partial
12 passing
Scanned 2 hours ago · 2 scans on record
Reach
38.5
/ 40
Whether AI crawlers and answer-engine fetchers are permitted to request the page at all, in robots.txt, in robots directives, and at the edge.
Readability
11.5
/ 25
Whether a fetcher that does not execute JavaScript receives the actual content, in markup an extractor can segment.
Structure
4
/ 20
Machine-readable markup that states the page's type, entities, canonical URL, and discrete facts instead of leaving them to be inferred.
Attribution
1
/ 15
Signals that let an answer engine name the author, date the content, resolve the publisher, and cite it under known terms.
Who is allowed to read this site 0 of 24 answer engines blocked
| Crawler | Operator | Uses content for | robots.txt | Live request |
|---|---|---|---|---|
| GPTBot Crawls content that may be used to train OpenAI's generative AI foundation models. |
OpenAI | Model training | allowed | served 200 |
| OAI-SearchBot Indexes pages so they can be surfaced and cited in ChatGPT search results, not for training. |
OpenAI | Answer index | allowed | served 200 |
| ChatGPT-User Fetches a page when a ChatGPT user or GPT Action asks for it; user-initiated, so robots rules may not apply. |
OpenAI | Live retrieval | allowed | not probed |
| OAI-AdsBot Visits pages submitted as ChatGPT ads to check policy compliance and ad relevance; not used for model training. |
OpenAI | Live retrieval | allowed | not probed |
| ClaudeBot Collects web content that may contribute to training Anthropic's models; honors Crawl-delay. |
Anthropic | Model training | allowed | served 200 |
| Claude-User Retrieves pages on demand when a Claude user's question needs live web content. |
Anthropic | Live retrieval | allowed | not probed |
| Claude-SearchBot Indexes content to improve the relevance and accuracy of Claude's search results. |
Anthropic | Answer index | allowed | not probed |
| anthropic-ai Legacy token widely blocked for Anthropic training; Anthropic now documents ClaudeBot, Claude-User and Claude-SearchBot. |
Anthropic | Model training | allowed | not probed |
| Google-Extended Control token with no user agent of its own; governs Gemini training and grounding use of Googlebot data. |
Model training | allowed | not probed | |
| Googlebot Crawls and renders pages for Google Search, Images, Video, News and Discover. |
Answer index | allowed | not probed | |
| Googlebot-News Robots token controlling Google News inclusion; crawling itself uses the Googlebot user agents. |
Answer index | allowed | not probed | |
| Google-CloudVertexBot Crawls sites at a site owner's request to build Vertex AI agents; no effect on Google Search. |
Live retrieval | allowed | not probed | |
| GoogleOther Generic Google crawler used by product teams for one-off fetches such as internal research and development. |
Model training | allowed | not probed | |
| Applebot Crawls for Siri, Spotlight and Safari search; falls back to Googlebot rules and ignores Crawl-delay. |
Apple | Answer index | allowed | not probed |
| Applebot-Extended Control token with no user agent; disallowing it excludes crawled content from Apple foundation model training. |
Apple | Model training | allowed | not probed |
| Bingbot Indexes pages for Bing search and the Copilot answers that are grounded in the Bing index. |
Microsoft | Answer index | allowed | not probed |
| msnbot Legacy Microsoft search crawler token still honored alongside bingbot. |
Microsoft | Answer index | allowed | not probed |
| PerplexityBot Indexes and links pages in Perplexity search results; not used to collect foundation model training data. |
Perplexity | Answer index | allowed | served 200 |
| Perplexity-User ignores robots Fetches a page for a specific user question; Perplexity documents that it generally ignores robots.txt. |
Perplexity | Live retrieval | allowed | not probed |
| Meta-ExternalAgent Crawls the web to train Meta's foundation AI models and to index content directly into products. |
Meta | Model training | allowed | not probed |
| Meta-ExternalFetcher ignores robots Fetches individual links for agentic AI tasks; Meta documents that it may bypass robots.txt. |
Meta | Live retrieval | allowed | not probed |
| FacebookBot Crawls public pages to improve language models behind Meta's speech recognition technology. |
Meta | Model training | allowed | not probed |
| Meta-WebIndexer Indexes pages so Meta AI can cite and link them in its search answers. |
Meta | Answer index | allowed | not probed |
| Meta-ExternalAds Crawls the web to improve Meta's advertising and other business products and services. |
Meta | Model training | allowed | not probed |
| facebookexternalhit ignores robots Fetches shared links for Facebook, Instagram and Messenger previews; may bypass robots.txt for integrity checks. |
Meta | Live retrieval | allowed | not probed |
| Bytespider ignores robots Downloads content to train ByteDance LLMs and is widely reported to ignore robots.txt directives. |
ByteDance | Model training | allowed | not probed |
| TikTokSpider ignores robots Fetches shared URLs for TikTok link previews and feeds; not expected to follow robots.txt. |
ByteDance | Live retrieval | allowed | not probed |
| Amazonbot Crawls for Amazon product and Alexa answers and may use the content to train Amazon AI models. |
Amazon | Model training | allowed | not probed |
| Amzn-SearchBot Indexes content for Amazon search experiences such as Alexa; does not crawl for generative AI training. |
Amazon | Answer index | allowed | not probed |
| Amzn-User ignores robots Fetches live pages to answer a user's Alexa question; Amazon documents it may not follow all robots.txt rules. |
Amazon | Live retrieval | allowed | not probed |
| CCBot Builds the open Common Crawl web archive, a common source of LLM pretraining corpora. |
Common Crawl Foundation | Archive | allowed | not probed |
| Diffbot Extracts structured page data for Diffbot's knowledge graph, which is licensed to AI customers. |
Diffbot | Model training | allowed | not probed |
| omgili Collects forum, news and blog content that Webz.io sells as web data feeds, including for AI training. |
Webz.io | Model training | allowed | not probed |
| omgilibot Legacy Omgili search crawler token still blocked alongside the current omgili agent. |
Webz.io | Model training | allowed | not probed |
| AI2Bot Collects web text for Ai2's open datasets used to train open language models such as OLMo. |
Allen Institute for AI | Model training | allowed | not probed |
| cohere-ai Retrieves pages to answer user-initiated prompts in Cohere's enterprise AI products. |
Cohere | Live retrieval | allowed | not probed |
| cohere-training-data-crawler Downloads training data for the large language models behind Cohere's enterprise AI products. |
Cohere | Model training | allowed | not probed |
| MistralAI-User Fetches pages on demand so Mistral's Vibe assistant can answer a question with live, cited web content. |
Mistral AI | Live retrieval | allowed | not probed |
| MistralAI-Index Indexes content for Mistral search behind Vibe answers; not used for generative AI training. |
Mistral AI | Answer index | allowed | not probed |
| MistralAI-Training Crawls web content to build datasets for training Mistral's generative AI models. |
Mistral AI | Model training | allowed | not probed |
| DuckAssistBot Crawls pages in real time for DuckDuckGo's cited AI-assisted answers; not used for model training. |
DuckDuckGo | Live retrieval | allowed | not probed |
| YouBot Indexes pages for You.com search results and the AI answers built on that index. |
You.com | Answer index | allowed | not probed |
| PanguBot Collects web content used to train Huawei's PanGu family of large models. |
Huawei | Model training | allowed | not probed |
| Timpibot Crawls pages for Timpi's decentralized index, which is also used as LLM training data. |
Timpi | Model training | allowed | not probed |
| ImagesiftBot Downloads public images plus surrounding text to build ImageSift's searchable image index. |
ImageSift (Hive) | Model training | allowed | not probed |
| Kangaroo Bot Scrapes site content into datasets used to train the Kangaroo LLM. |
Kangaroo LLM | Model training | allowed | not probed |
| SemrushBot-OCOB Crawls pages to feed Semrush's ContentShake AI writing tool. |
Semrush | Model training | allowed | not probed |
| Scrapy Generic scraping framework often used to build AI training datasets; obeys robots.txt only when ROBOTSTXT_OBEY is on. |
Zyte (open-source framework) | Model training | allowed | not probed |
Two of these columns matter differently. robots.txt is what the site declares. Live request is what actually happened when we sent a real request using that crawler's user agent from a datacentre IP, which is how edge blocking, rate limits and challenge pages show up even when robots.txt looks permissive.
Reach 38.5 / 40
!
No usable robots.txt at /robots.txt
No robots.txt is served. Everything is crawlable by default, but you cannot declare a sitemap or set per-crawler rules.
Why it matters. Every compliant crawler requests /robots.txt before any other path. A 5xx or connection error there is treated as a temporary site-wide disallow by Googlebot and OpenAI's fetchers, so the whole origin stops being crawled until it recovers.
Fix. Serve `/robots.txt` at the apex of every hostname you want crawled, with HTTP 200 and `Content-Type: text/plain`. Never answer that path with a 5xx, a redirect chain to HTML, or a soft-404 page rendered by the app router. Keep the file under 500 KiB, since parsers truncate past that limit. Add a `Sitemap:` line so discovery does not depend on link crawling alone.
Fix. Serve `/robots.txt` at the apex of every hostname you want crawled, with HTTP 200 and `Content-Type: text/plain`. Never answer that path with a 5xx, a redirect chain to HTML, or a soft-404 page rendered by the app router. Keep the file under 500 KiB, since parsers truncate past that limit. Add a `Sitemap:` line so discovery does not depend on link crawling alone.
User-agent: *
Allow: /
Disallow: /cart
Disallow: /checkout
Sitemap: https://example.com/sitemap.xml3 pt
✓
Answer-engine fetchers are allowed to retrieve and cite this page
All 24 answer-engine fetchers are allowed to retrieve pages for citation.
12 pt
✓
Training crawlers may fetch this path
All 23 tracked training crawlers are allowed.
8 pt
✓
AI user agents receive the same 200 response as browsers
Live requests as 4 AI user agents were served normally.
6 pt
✓
No blanket disallow applies to this path
The wildcard group does not disallow the entire site.
5 pt
✓
No Crawl-delay directive constrains fetchers
No Crawl-delay directive.
2 pt
✓
Page is indexable, with no noindex directive
No noindex directive on the homepage.
2 pt
✓
Full-length snippet extraction is permitted
Snippets are not restricted by meta tags.
1 pt
✓
X-Robots-Tag header is absent or permissive
No restrictive X-Robots-Tag header.
1 pt
Readability 11.5 / 25
✗
No <main> or <article> element marks the primary content
No <main> or <article> element, so extractors must guess where the content starts.
Why it matters. Boilerplate removers use `<main>` and `<article>` to decide which subtree is the content and which is navigation, promo, and footer. Without a landmark the extractor guesses by text density and often keeps the nav while dropping part of the body.
Fix. Wrap the page's unique content in exactly one `<main>` element, and use `<article>` for each self-contained item inside it. Put navigation in `<nav>`, site chrome in `<header>` and `<footer>`, and tangential blocks in `<aside>` so they are cleanly separable. Do not nest the content inside a `<div>` whose only meaning is a CSS class.
Fix. Wrap the page's unique content in exactly one `<main>` element, and use `<article>` for each self-contained item inside it. Put navigation in `<nav>`, site chrome in `<header>` and `<footer>`, and tangential blocks in `<aside>` so they are cleanly separable. Do not nest the content inside a `<div>` whose only meaning is a CSS class.
<body>
<header><nav><!-- site navigation --></nav></header>
<main>
<article>
<h1>How answer engines fetch your pages</h1>
<p>Content that should be extracted and quoted.</p>
</article>
</main>
<aside><!-- related links --></aside>
<footer><!-- legal, contact --></footer>
</body>3 pt
✗
Heading levels are missing, duplicated, or skipped
0 H1 and 0 headings total.
Why it matters. Chunkers split long pages on heading boundaries and carry the nearest heading into each chunk's metadata. A page with no `h1`, several competing `h1`s, or levels that skip from `h2` to `h4` produces chunks whose topic labels do not match their text, so retrieval matches the wrong passage.
Fix. Give every page one `h1` that names its subject, then nest `h2` and `h3` without skipping levels. Make each heading describe the section beneath it in words a reader would search for, rather than a label like "Overview". Style with CSS instead of choosing heading levels for their font size, and never use a heading tag for a caption or a button.
Fix. Give every page one `h1` that names its subject, then nest `h2` and `h3` without skipping levels. Make each heading describe the section beneath it in words a reader would search for, rather than a label like "Overview". Style with CSS instead of choosing heading levels for their font size, and never use a heading tag for a caption or a button.
<h1>Robots.txt rules for AI crawlers</h1>
<h2>Training crawlers</h2>
<h3>GPTBot</h3>
<h3>ClaudeBot</h3>
<h2>Retrieval and citation fetchers</h2>
<h3>OAI-SearchBot</h3>3 pt
✗
No meta description summarises the page
No meta description, so answer engines invent their own summary.
Why it matters. The meta description is a short, author-written summary that retrieval systems index alongside the body and often surface as the preview line beside a citation. Without it the surface generates a summary from whatever fragment it extracted, which may be navigation text.
Fix. Add a `<meta name="description">` of roughly 110 to 160 characters that states what the page answers, written per page rather than per template. Include the concrete nouns a reader would search for, and do not restate the title word for word. Leave it out entirely rather than shipping the same string sitewide.
Fix. Add a `<meta name="description">` of roughly 110 to 160 characters that states what the page answers, written per page rather than per template. Include the concrete nouns a reader would search for, and do not restate the title word for word. Leave it out entirely rather than shipping the same string sitewide.
<meta name="description" content="Which robots.txt rules block AI training crawlers, which block answer-engine fetchers, and how to allow citation while opting out of training." />2 pt
✗
No lang attribute declares the document language
No lang attribute on <html>.
Why it matters. `<html lang>` tells tokenizers and language filters which language the text is in. Without it a retrieval pipeline guesses, and a wrong guess drops the page from language-scoped candidate sets and degrades sentence splitting.
Fix. Set a valid BCP 47 tag on the root element, such as `lang="en"` or `lang="pt-BR"`. Mark inline passages in another language with `lang` on the containing element. If you publish translations, pair the declaration with `hreflang` alternates so each version is attributed to the right locale.
Fix. Set a valid BCP 47 tag on the root element, such as `lang="en"` or `lang="pt-BR"`. Mark inline passages in another language with `lang` on the containing element. If you publish translations, pair the declaration with `hreflang` alternates so each version is attributed to the right locale.
<html lang="en">
<head>
<link rel="alternate" hreflang="es" href="https://example.com/es/page" />
</head>
</html>1 pt
!
Raw HTML contains almost no readable text before JavaScript runs
277 words of text are present in the raw HTML. Most AI fetchers do not run JavaScript.
Why it matters. Most AI fetchers, including OAI-SearchBot, ChatGPT-User, Claude-User, and PerplexityBot, take the raw HTTP response and do not run JavaScript. A client-rendered page therefore arrives as an empty root element plus script tags, and the model has nothing to quote or summarise.
Fix. Render the primary content on the server: server-side rendering, static generation at build time, or prerendering into the HTML response. Keep client JavaScript for interactive extras such as filters and comments, never for the article body, product description, or specifications table. Verify by fetching the page with an AI user agent, stripping tags, and counting words, which is what the fetcher effectively sees. Confirm the numbers match the browser view rather than trusting a framework's hydration claim.
Fix. Render the primary content on the server: server-side rendering, static generation at build time, or prerendering into the HTML response. Keep client JavaScript for interactive extras such as filters and comments, never for the article body, product description, or specifications table. Verify by fetching the page with an AI user agent, stripping tags, and counting words, which is what the fetcher effectively sees. Confirm the numbers match the browser view rather than trusting a framework's hydration claim.
curl -s -A "OAI-SearchBot/1.0" https://example.com/page \
| sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' \
| tr -s '[:space:]' ' ' \
| wc -w9 pt
✓
Visible text makes up a healthy share of the HTML payload
Text is 39.0% of the 5 KB document; 0 KB is inline script.
4 pt
✓
Title is unique and describes the page in specific terms
Title is 33 characters: ". . .. e s x d o s . o r g .. . ."
3 pt
Structure 4 / 20
✗
No JSON-LD structured data found on the page
No JSON-LD structured data on the homepage.
Why it matters. JSON-LD gives an answer engine typed facts (headline, author, dates, price, publisher) without inferring them from prose, and those fields populate the entity record a citation is attached to. With no structured data every attribute has to be guessed from text, and guesses are dropped when confidence is low.
Fix. Add one `<script type="application/ld+json">` block describing the page's primary entity, using the schema.org type that actually fits: `Article` or `NewsArticle`, `Product`, `Recipe`, `Event`, `FAQPage`, or `SoftwareApplication`. Populate the required properties for that type and make every value match visible page content. Prefer JSON-LD over microdata or RDFa, since it is the format the major crawlers document, and render it server-side so non-JavaScript fetchers see it.
Fix. Add one `<script type="application/ld+json">` block describing the page's primary entity, using the schema.org type that actually fits: `Article` or `NewsArticle`, `Product`, `Recipe`, `Event`, `FAQPage`, or `SoftwareApplication`. Populate the required properties for that type and make every value match visible page content. Prefer JSON-LD over microdata or RDFa, since it is the format the major crawlers document, and render it server-side so non-JavaScript fetchers see it.
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Allow AI answer engines in robots.txt",
"description": "How to opt out of model training while staying citable.",
"url": "https://example.com/blog/robots-for-ai",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://example.com/blog/robots-for-ai"
},
"image": "https://example.com/images/robots-for-ai.png",
"inLanguage": "en",
"datePublished": "2026-02-11T09:00:00-05:00",
"dateModified": "2026-08-04T14:20:00-04:00",
"author": {
"@type": "Person",
"name": "Dana Reyes",
"url": "https://example.com/authors/dana-reyes"
},
"publisher": {
"@type": "Organization",
"name": "Example",
"url": "https://example.com",
"logo": {
"@type": "ImageObject",
"url": "https://example.com/logo.png",
"width": 512,
"height": 512
}
}
}7 pt
✗
Structured data is too generic for what the page is about
No entity types that answer engines consume.
Why it matters. Answer engines route by entity type: a `Product` node supplies price and availability, an `Article` node supplies author and dates, and a `FAQPage` node supplies question and answer pairs. A generic `WebPage` or `WebSite` node on a product or article page carries none of those fields, so the specific facts stay unavailable.
Fix. Replace bare `WebPage` and `WebSite` nodes with the most specific type that describes the page, and fill the properties that type defines. Use `@graph` to publish several linked nodes on one page, such as an `Article` whose `publisher` points at an `Organization` node by `@id`. Add `BreadcrumbList` for hierarchy and reuse the same `@id` values across pages so the entity resolves to one record.
Fix. Replace bare `WebPage` and `WebSite` nodes with the most specific type that describes the page, and fill the properties that type defines. Use `@graph` to publish several linked nodes on one page, such as an `Article` whose `publisher` points at an `Organization` node by `@id`. Add `BreadcrumbList` for hierarchy and reuse the same `@id` values across pages so the entity resolves to one record.
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Product",
"@id": "https://example.com/products/widget#product",
"name": "Widget Pro",
"sku": "WGT-PRO-1",
"brand": { "@type": "Brand", "name": "Example" },
"offers": {
"@type": "Offer",
"url": "https://example.com/products/widget",
"price": "49.00",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock"
}
},
{
"@type": "BreadcrumbList",
"itemListElement": [
{ "@type": "ListItem", "position": 1, "name": "Products", "item": "https://example.com/products" },
{ "@type": "ListItem", "position": 2, "name": "Widget Pro" }
]
}
]
}4 pt
✗
No canonical URL is declared for this page
No rel=canonical, so duplicate URLs split citation signals.
Why it matters. Tracking parameters, trailing slashes, and http and https variants make one document reachable at many URLs. Without `rel=canonical` the retrieval index can hold several near-duplicate records, splitting the signals that rank the page and making the cited link unstable.
Fix. Emit an absolute, self-referential `<link rel="canonical">` on every page, using the exact scheme, host, and path you want cited. Point parameterised and paginated variants at the canonical document, and make sure the canonical URL itself returns 200 rather than redirecting. Keep the canonical, the `og:url`, the sitemap entry, and the JSON-LD `url` identical, since conflicting values are treated as a weak hint and may be overridden.
Fix. Emit an absolute, self-referential `<link rel="canonical">` on every page, using the exact scheme, host, and path you want cited. Point parameterised and paginated variants at the canonical document, and make sure the canonical URL itself returns 200 rather than redirecting. Keep the canonical, the `og:url`, the sitemap entry, and the JSON-LD `url` identical, since conflicting values are treated as a weak hint and may be overridden.
<link rel="canonical" href="https://example.com/blog/robots-for-ai" />
<meta property="og:url" content="https://example.com/blog/robots-for-ai" />3 pt
✗
No reachable XML sitemap is declared
No sitemap.xml and none declared in robots.txt.
Why it matters. A sitemap gives crawlers the URL list and `lastmod` timestamps directly, instead of leaving discovery to link traversal that never reaches pages behind search forms or JavaScript routers. Fetchers use `lastmod` to prioritise recrawls, so fresh content is picked up sooner.
Fix. Publish a sitemap of canonical, indexable URLs and reference it with an absolute `Sitemap:` line in robots.txt. Keep each file under 50,000 URLs and 50 MiB uncompressed, using a sitemap index when you exceed either limit. Set `lastmod` from real content changes rather than the build clock, and exclude redirects, error pages, and non-canonical variants.
Fix. Publish a sitemap of canonical, indexable URLs and reference it with an absolute `Sitemap:` line in robots.txt. Keep each file under 50,000 URLs and 50 MiB uncompressed, using a sitemap index when you exceed either limit. Set `lastmod` from real content changes rather than the build clock, and exclude redirects, error pages, and non-canonical variants.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/blog/robots-for-ai</loc>
<lastmod>2026-08-04T14:20:00-04:00</lastmod>
</url>
</urlset>2 pt
✓
JSON-LD parses cleanly with recognised schema.org terms
All JSON-LD blocks parse cleanly.
3 pt
✓
Key facts are available in lists or tables
4 tables, 1 lists, 0 code blocks, 0 question headings.
1 pt
Attribution 1 / 15
✗
No /llms.txt index of canonical pages
No /llms.txt.
Why it matters. `/llms.txt` is a markdown file that points an assistant at the canonical pages for a site, so retrieval does not depend on which page a search happened to return. Its format is fixed: one `#` title, a `>` blockquote summary, then `##` sections of markdown links with short notes.
Fix. Publish `/llms.txt` as `text/plain` markdown: an `#` H1 with the project name, a `>` blockquote summary, optional plain paragraphs of context, then `##` sections whose bullets are `[title](absolute-url): note`. Link the pages you want quoted, put lower-priority links under an `## Optional` section, and prefer URLs that also serve clean markdown. Keep it generated from the same source as your sitemap so it does not drift, and remember it is a hint for assistants, not an access control mechanism.
Fix. Publish `/llms.txt` as `text/plain` markdown: an `#` H1 with the project name, a `>` blockquote summary, optional plain paragraphs of context, then `##` sections whose bullets are `[title](absolute-url): note`. Link the pages you want quoted, put lower-priority links under an `## Optional` section, and prefer URLs that also serve clean markdown. Keep it generated from the same source as your sitemap so it does not drift, and remember it is a hint for assistants, not an access control mechanism.
# Example
> Example publishes reference documentation for the Widget API and guides for
> configuring crawler access.
Prefer the pages below over search results; each URL is canonical.
## Docs
- [Widget API reference](https://example.com/docs/api): endpoints, auth, limits.
- [Quickstart](https://example.com/docs/quickstart): first request in five minutes.
## Policies
- [Crawler policy](https://example.com/legal/crawlers): which agents we allow.
## Optional
- [Changelog](https://example.com/changelog): dated release notes.4 pt
✗
No machine-readable author is attached to the page
No author or Person entity, which weakens the authority signals answer engines use.
Why it matters. An `author` property in structured data is what lets an answer engine name a person or organisation as the source and link the byline to a stable profile. A byline that exists only as styled text is not reliably associated with the document during extraction.
Fix. Add an `author` property to the page's `Article`, `BlogPosting`, or `NewsArticle` node, typed as `Person` or `Organization`, with a `name` and a `url` pointing at a real profile page. Give each author a stable `@id` and reuse it across posts so the entity consolidates. Keep the visible byline identical to the structured value, and avoid generic names such as "Admin" or "Staff Writer" where a real author exists.
Fix. Add an `author` property to the page's `Article`, `BlogPosting`, or `NewsArticle` node, typed as `Person` or `Organization`, with a `name` and a `url` pointing at a real profile page. Give each author a stable `@id` and reuse it across posts so the entity consolidates. Keep the visible byline identical to the structured value, and avoid generic names such as "Admin" or "Staff Writer" where a real author exists.
{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Allow AI answer engines in robots.txt",
"author": {
"@type": "Person",
"@id": "https://example.com/authors/dana-reyes#person",
"name": "Dana Reyes",
"url": "https://example.com/authors/dana-reyes",
"jobTitle": "Infrastructure Engineer",
"sameAs": ["https://github.com/danareyes"]
}
}3 pt
✗
No machine-readable published or modified date
No publication or modification dates in structured data.
Why it matters. Answer engines prefer recent sources for questions about current state and use `dateModified` to decide whether a cached copy needs refetching. With no machine-readable date the page is treated as undated and loses to competitors that publish one.
Fix. Publish `datePublished` and `dateModified` in the page's structured data as ISO 8601 values with a timezone offset. Update `dateModified` only when the content actually changes, since bumping it on every deploy trains crawlers to ignore it. Mirror the value in a visible `<time datetime>` element so the rendered text and the metadata agree, and keep the sitemap `lastmod` consistent with it.
Fix. Publish `datePublished` and `dateModified` in the page's structured data as ISO 8601 values with a timezone offset. Update `dateModified` only when the content actually changes, since bumping it on every deploy trains crawlers to ignore it. Mirror the value in a visible `<time datetime>` element so the rendered text and the metadata agree, and keep the sitemap `lastmod` consistent with it.
<time datetime="2026-08-04T14:20:00-04:00">Updated August 4, 2026</time>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Allow AI answer engines in robots.txt",
"datePublished": "2026-02-11T09:00:00-05:00",
"dateModified": "2026-08-04T14:20:00-04:00"
}
</script>3 pt
✗
No Organization entity identifies the publisher
No Organization entity, so the brand is harder to resolve to a known entity.
Why it matters. An `Organization` node with `sameAs` links resolves your site to a single entity across knowledge graphs instead of leaving the publisher name as an ambiguous string. That resolution is what lets an assistant attribute an answer to your brand and reuse your name, logo, and contact details.
Fix. Publish one `Organization` node, usually on the home page, with `name`, `url`, `logo`, and `sameAs` pointing at the profiles that already describe you: Wikipedia or Wikidata, Crunchbase, LinkedIn, and your primary social accounts. Give it a stable `@id` such as `https://example.com/#organization` and reference that `@id` from each page's `publisher` property instead of repeating the block. Add `contactPoint` and `address` when they are public, and keep every value identical to what the site shows.
Fix. Publish one `Organization` node, usually on the home page, with `name`, `url`, `logo`, and `sameAs` pointing at the profiles that already describe you: Wikipedia or Wikidata, Crunchbase, LinkedIn, and your primary social accounts. Give it a stable `@id` such as `https://example.com/#organization` and reference that `@id` from each page's `publisher` property instead of repeating the block. Add `contactPoint` and `address` when they are public, and keep every value identical to what the site shows.
{
"@context": "https://schema.org",
"@type": "Organization",
"@id": "https://example.com/#organization",
"name": "Example",
"legalName": "Example Holdings, Inc.",
"url": "https://example.com",
"logo": {
"@type": "ImageObject",
"url": "https://example.com/logo.png",
"width": 512,
"height": 512
},
"sameAs": [
"https://www.wikidata.org/wiki/Q00000000",
"https://www.linkedin.com/company/example",
"https://github.com/example"
],
"contactPoint": {
"@type": "ContactPoint",
"contactType": "customer support",
"email": "support@example.com",
"areaServed": "US",
"availableLanguage": ["en"]
}
}3 pt
!
No machine-readable license or usage terms for the content
No licence declaration, so reuse terms are ambiguous.
Why it matters. A `license` property, or a linked terms page, states the reuse conditions in a place a crawler can read, rather than leaving them to be inferred. Where terms are unstated, some pipelines default to the more restrictive handling, which reduces how much of the text is quoted.
Fix. Add a `license` property to the page's structured data pointing at a specific license URL, such as a Creative Commons deed or your own terms page, and add `rel="license"` on the visible link. Use `usageInfo` for conditions that are not a standard license, such as attribution wording or an API-only clause. State the terms once, at a stable URL, and reference it from every page rather than restating it per template.
Fix. Add a `license` property to the page's structured data pointing at a specific license URL, such as a Creative Commons deed or your own terms page, and add `rel="license"` on the visible link. Use `usageInfo` for conditions that are not a standard license, such as attribution wording or an API-only clause. State the terms once, at a stable URL, and reference it from every page rather than restating it per template.
<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Article",
"headline": "Allow AI answer engines in robots.txt",
"license": "https://creativecommons.org/licenses/by/4.0/",
"usageInfo": "https://example.com/legal/content-reuse",
"creditText": "Example, crawlcensus research desk"
}
</script>2 pt
Evidence raw measurements
robots.txt
No robots.txt is served at /robots.txt.
Machine-readable extras
- llms.txt
- absent
- ai.txt
- absent
- Sitemap
- absent
- Feeds
- none
- Schema types
- none
- Edge
- Cloudflare
- Final URL
- https://esxdos.org/index.html
- HTML size
- 5 KB, 277 words of text
Publish the score free
Sites that score well embed the badge. It links back to this live report, which re-measures on every scan.
<a href="https://crawlcensus.com/site/esxdos.org"><img src="https://crawlcensus.com/badge/esxdos.org.svg" alt="AI access score for esxdos.org" width="150" height="20"></a>Track changes on this domain pro
Crawler policy is edited quietly. We re-scan monitored domains daily, keep the history, and email you the moment a crawler is blocked or unblocked, an llms.txt appears, or the score moves.
One address, unlimited domains during the beta. No newsletter, only change alerts.