cnn.com

Why it matters. Retrieval fetchers (OAI-SearchBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User) load a page at answer time to quote and link it. When robots.txt disallows them the assistant drops the URL from its candidate set, so the citation goes to a competitor while your training exposure stays exactly the same.
Fix. Separate the two crawler classes in robots.txt instead of blocking by vendor. Allow `OAI-SearchBot` and `ChatGPT-User` (OpenAI retrieval and user-initiated fetches), `Claude-SearchBot` and `Claude-User` (Anthropic retrieval), and `PerplexityBot` and `Perplexity-User`; keep any opt-out you want on `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, and `Applebot-Extended`. Order does not decide precedence in RFC 9309 parsers, the longest matching rule does, so make the allow rules at least as specific as the disallow rules. Blocking retrieval buys nothing on training, because the training crawlers are separate user agents with separate rules.

# Retrieval and citation fetchers: allowed.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Training and dataset crawlers: disallowed.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /
Disallow: /checkout

Sitemap: https://example.com/sitemap.xml

Reference

Why it matters. A permissive robots.txt is irrelevant if the edge answers an AI user agent with 403, 429, or a JavaScript challenge. Retrieval fetchers do not solve interstitials, so the assistant records a fetch failure and answers from another source.
Fix. Fetch the page with each AI user agent string and compare the status and byte count against a browser request. On Cloudflare, check whether the "Block AI bots" toggle in AI Crawl Control, a Bot Fight Mode rule, or a WAF custom rule on `cf.verified_bot_category` is catching the request, then narrow it: block the training category and add a skip rule for the retrieval agents you want citing you. Verified bots must not be handed Managed Challenge, since a challenge is a hard failure for a non-browser client. Re-test after every WAF or bot-management change, because these toggles are applied zone-wide.

for ua in "OAI-SearchBot/1.0" "ChatGPT-User/1.0" "Claude-User/1.0" \
          "Claude-SearchBot/1.0" "PerplexityBot/1.0" "Mozilla/5.0"; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/)
  printf '%s\t%s\n' "$code" "$ua"
done

Reference

Why it matters. GPTBot, ClaudeBot, and CCBot collect pages into pretraining corpora, and Google-Extended and Applebot-Extended are opt-out tokens that govern whether already-crawled pages may be used for Gemini and Apple Intelligence. Disallowing them removes your text from the corpora models generalise from, which is a policy choice, not a bug.
Fix. Decide this deliberately rather than by inheriting a template. If you want your content in model weights, remove the `Disallow: /` groups for `GPTBot`, `ClaudeBot`, `CCBot`, `Google-Extended`, and `Applebot-Extended`. If you do not, keep those groups but scope them to the paths that matter and leave retrieval fetchers untouched, because `Google-Extended` and `Applebot-Extended` only control training use and never affect search or answer citation. Record the decision somewhere durable so the next robots.txt edit does not silently reverse it.

# Opt out of model training only.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Reference

Why it matters. Most AI fetchers, including OAI-SearchBot, ChatGPT-User, Claude-User, and PerplexityBot, take the raw HTTP response and do not run JavaScript. A client-rendered page therefore arrives as an empty root element plus script tags, and the model has nothing to quote or summarise.
Fix. Render the primary content on the server: server-side rendering, static generation at build time, or prerendering into the HTML response. Keep client JavaScript for interactive extras such as filters and comments, never for the article body, product description, or specifications table. Verify by fetching the page with an AI user agent, stripping tags, and counting words, which is what the fetcher effectively sees. Confirm the numbers match the browser view rather than trusting a framework's hydration claim.

curl -s -A "OAI-SearchBot/1.0" https://example.com/page \
  | sed -e 's/<script[^>]*>.*<\/script>//g' -e 's/<[^>]*>/ /g' \
  | tr -s '[:space:]' ' ' \
  | wc -w

Reference

Why it matters. Extractors strip scripts, styles, and wrapper markup before passing text to a model, and a page where content is a small fraction of the payload loses more of it to boilerplate removal. Inline JSON state blobs and deeply nested wrappers also push real text past the truncation limit of a retrieval context.
Fix. Move inline hydration state and large inline scripts out of the document, or fetch them after load instead of embedding them. Flatten wrapper `div` trees and let semantic elements carry the content, and keep utility-class soup out of the article body. Serve the same text without the boilerplate at a stable URL if you need a clean extraction target.

Why it matters. Boilerplate removers use `<main>` and `<article>` to decide which subtree is the content and which is navigation, promo, and footer. Without a landmark the extractor guesses by text density and often keeps the nav while dropping part of the body.
Fix. Wrap the page's unique content in exactly one `<main>` element, and use `<article>` for each self-contained item inside it. Put navigation in `<nav>`, site chrome in `<header>` and `<footer>`, and tangential blocks in `<aside>` so they are cleanly separable. Do not nest the content inside a `<div>` whose only meaning is a CSS class.

<body>
  <header><nav><!-- site navigation --></nav></header>
  <main>
    <article>
      <h1>How answer engines fetch your pages</h1>
      <p>Content that should be extracted and quoted.</p>
    </article>
  </main>
  <aside><!-- related links --></aside>
  <footer><!-- legal, contact --></footer>
</body>

Reference

Why it matters. Chunkers split long pages on heading boundaries and carry the nearest heading into each chunk's metadata. A page with no `h1`, several competing `h1`s, or levels that skip from `h2` to `h4` produces chunks whose topic labels do not match their text, so retrieval matches the wrong passage.
Fix. Give every page one `h1` that names its subject, then nest `h2` and `h3` without skipping levels. Make each heading describe the section beneath it in words a reader would search for, rather than a label like "Overview". Style with CSS instead of choosing heading levels for their font size, and never use a heading tag for a caption or a button.

<h1>Robots.txt rules for AI crawlers</h1>
<h2>Training crawlers</h2>
<h3>GPTBot</h3>
<h3>ClaudeBot</h3>
<h2>Retrieval and citation fetchers</h2>
<h3>OAI-SearchBot</h3>

Reference

Why it matters. Lists, tables with header cells, and definition lists survive HTML-to-text conversion as discrete rows a model can lift into an answer verbatim. The same facts written as a paragraph, or laid out with positioned `div`s, lose their row and column relationships during extraction.
Fix. Put specifications, comparisons, pricing, and steps into real `<table>` markup with `<caption>` and `<th scope>`, or into `<ul>`, `<ol>`, and `<dl>` elements. Do not simulate tables with `div` grids, and avoid images of tables, which carry no extractable text. Keep one fact per row or list item so a chunk stays meaningful on its own.

<table>
  <caption>AI fetcher purposes</caption>
  <thead>
    <tr><th scope="col">User agent</th><th scope="col">Purpose</th></tr>
  </thead>
  <tbody>
    <tr><td>GPTBot</td><td>Model training</td></tr>
    <tr><td>OAI-SearchBot</td><td>Retrieval and citation</td></tr>
  </tbody>
</table>

Reference

Why it matters. Answer engines route by entity type: a `Product` node supplies price and availability, an `Article` node supplies author and dates, and a `FAQPage` node supplies question and answer pairs. A generic `WebPage` or `WebSite` node on a product or article page carries none of those fields, so the specific facts stay unavailable.
Fix. Replace bare `WebPage` and `WebSite` nodes with the most specific type that describes the page, and fill the properties that type defines. Use `@graph` to publish several linked nodes on one page, such as an `Article` whose `publisher` points at an `Organization` node by `@id`. Add `BreadcrumbList` for hierarchy and reuse the same `@id` values across pages so the entity resolves to one record.

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Product",
      "@id": "https://example.com/products/widget#product",
      "name": "Widget Pro",
      "sku": "WGT-PRO-1",
      "brand": { "@type": "Brand", "name": "Example" },
      "offers": {
        "@type": "Offer",
        "url": "https://example.com/products/widget",
        "price": "49.00",
        "priceCurrency": "USD",
        "availability": "https://schema.org/InStock"
      }
    },
    {
      "@type": "BreadcrumbList",
      "itemListElement": [
        { "@type": "ListItem", "position": 1, "name": "Products", "item": "https://example.com/products" },
        { "@type": "ListItem", "position": 2, "name": "Widget Pro" }
      ]
    }
  ]
}

Reference

Why it matters. `/llms.txt` is a markdown file that points an assistant at the canonical pages for a site, so retrieval does not depend on which page a search happened to return. Its format is fixed: one `#` title, a `>` blockquote summary, then `##` sections of markdown links with short notes.
Fix. Publish `/llms.txt` as `text/plain` markdown: an `#` H1 with the project name, a `>` blockquote summary, optional plain paragraphs of context, then `##` sections whose bullets are `[title](absolute-url): note`. Link the pages you want quoted, put lower-priority links under an `## Optional` section, and prefer URLs that also serve clean markdown. Keep it generated from the same source as your sitemap so it does not drift, and remember it is a hint for assistants, not an access control mechanism.

# Example

> Example publishes reference documentation for the Widget API and guides for
> configuring crawler access.

Prefer the pages below over search results; each URL is canonical.

## Docs

- [Widget API reference](https://example.com/docs/api): endpoints, auth, limits.
- [Quickstart](https://example.com/docs/quickstart): first request in five minutes.

## Policies

- [Crawler policy](https://example.com/legal/crawlers): which agents we allow.

## Optional

- [Changelog](https://example.com/changelog): dated release notes.

Reference

Why it matters. An `author` property in structured data is what lets an answer engine name a person or organisation as the source and link the byline to a stable profile. A byline that exists only as styled text is not reliably associated with the document during extraction.
Fix. Add an `author` property to the page's `Article`, `BlogPosting`, or `NewsArticle` node, typed as `Person` or `Organization`, with a `name` and a `url` pointing at a real profile page. Give each author a stable `@id` and reuse it across posts so the entity consolidates. Keep the visible byline identical to the structured value, and avoid generic names such as "Admin" or "Staff Writer" where a real author exists.

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Allow AI answer engines in robots.txt",
  "author": {
    "@type": "Person",
    "@id": "https://example.com/authors/dana-reyes#person",
    "name": "Dana Reyes",
    "url": "https://example.com/authors/dana-reyes",
    "jobTitle": "Infrastructure Engineer",
    "sameAs": ["https://github.com/danareyes"]
  }
}

Reference

Why it matters. Answer engines prefer recent sources for questions about current state and use `dateModified` to decide whether a cached copy needs refetching. With no machine-readable date the page is treated as undated and loses to competitors that publish one.
Fix. Publish `datePublished` and `dateModified` in the page's structured data as ISO 8601 values with a timezone offset. Update `dateModified` only when the content actually changes, since bumping it on every deploy trains crawlers to ignore it. Mirror the value in a visible `<time datetime>` element so the rendered text and the metadata agree, and keep the sitemap `lastmod` consistent with it.

<time datetime="2026-08-04T14:20:00-04:00">Updated August 4, 2026</time>

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Allow AI answer engines in robots.txt",
  "datePublished": "2026-02-11T09:00:00-05:00",
  "dateModified": "2026-08-04T14:20:00-04:00"
}
</script>

Reference

Why it matters. A `license` property, or a linked terms page, states the reuse conditions in a place a crawler can read, rather than leaving them to be inferred. Where terms are unstated, some pipelines default to the more restrictive handling, which reduces how much of the text is quoted.
Fix. Add a `license` property to the page's structured data pointing at a specific license URL, such as a Creative Commons deed or your own terms page, and add `rel="license"` on the visible link. Use `usageInfo` for conditions that are not a standard license, such as attribution wording or an API-only clause. State the terms once, at a stable URL, and reference it from every page rather than restating it per template.

<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a>

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Allow AI answer engines in robots.txt",
  "license": "https://creativecommons.org/licenses/by/4.0/",
  "usageInfo": "https://example.com/legal/content-reuse",
  "creditText": "Example, crawlcensus research desk"
}
</script>

Reference