robots.txt for AI crawlers: the complete reference

A compliant crawler resolves your robots.txt in three steps: pick exactly one group by user-agent token, pick the winning rule inside that group by pattern length, and if nothing matched, fetch. That is the whole algorithm. Almost every robots.txt failure we see in scans comes from assuming a fourth step exists.

The specification is RFC 9309, which the major AI crawler operators say they implement: OpenAI, Anthropic, Google, Apple, Perplexity, Amazon, Mistral, Common Crawl and DuckDuckGo all publish robots.txt tokens and describe standard behaviour. What follows is the mechanism, then the failure modes, then a reference file you can adapt.

Step one: group selection

Consecutive User-agent lines with no rules between them form a single group that shares the rules beneath it. A crawler compares its own product token against every group's tokens and selects one group:

1. Longest matching token wins. The group whose token is the longest case-insensitive match for the crawler's product token is selected. A crawler identifying as Googlebot-News obeys a googlebot-news group even when a googlebot group also exists in the same file.
2. Otherwise the * group. With no token match at all, the crawler uses the User-agent: * group.
3. Otherwise everything is allowed. No matching group and no * group means no restrictions. Allow-by-default is the specification's position, not a lenient interpretation of it.

Two operators complicate this deliberately. Apple documents that "if robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions" (support.apple.com/en-us/119829). Amazon documents the same fallback for Amzn-SearchBot toward other search-bot rules (developer.amazon.com/amazonbot). For these two agents, the effective policy is not readable from their own group alone.

Step two: rule precedence inside the group

Every Allow and Disallow in the selected group is matched against the request path. The rule with the longest pattern decides. When an Allow and a Disallow match with equal pattern length, Allow wins. Two wildcards are supported: * matches any sequence of characters, and a trailing $ anchors the pattern to the end of the path.

The last row is worth its own sentence. An empty Disallow: is a valid line that matches no path, so it permits everything in its group. It is the specification's idiom for "this agent is unrestricted", and it is not a typo to be repaired. Deleting the value from a working Disallow: / inverts the rule.

Four mistakes that void an otherwise correct file

1. A blanket rule that shadows nothing

The pattern we see most often is a file that ends with User-agent: * and Disallow: /, written by someone who believed it was a catch-all, sitting in a file that also contains named groups for Googlebot and bingbot. The named crawlers are unaffected. Every crawler without a group is fully blocked, and that set includes most answer-engine retrieval fetchers and, by construction, every agent that ships after you wrote the file. The blast radius is exactly the agents you did not think about.

2. Blocking by path when the crawler enters by sitemap

Path rules govern fetching, not discovery. A URL listed in your sitemap, linked from a canonical tag, or already held in a search index is still a candidate; the crawler simply gets a disallow when it tries to fetch it. The observable result is a URL that answer engines know about and cannot read, which is worse than either being open or being absent. If a section is genuinely private, remove it from the sitemap and from internal links as well as disallowing it. If it is public, do not disallow it because a staging convention leaked into production.

3. Case handling

Tokens and field names are case-insensitive; paths are not. Disallow: /admin/ leaves /Admin/ open. On servers that resolve both to the same handler this is a bypass, not a nuance.

4. Assuming one file covers the property

robots.txt is fetched at host level. Amazon's documentation states the rule explicitly: site.example.com/robots.txt governs that host independently of example.com. Marketing site, docs, help centre, status page and shop each need their own file. A scan of your apex domain tells you nothing about your docs subdomain, which is frequently the content an assistant most wants to quote.

Crawl-delay: who honours it

Crawl-delay is not in RFC 9309, and support is genuinely split:

The practical consequence: a Crawl-delay intended to protect a fragile origin protects it from some crawlers and not others, while a delay long enough to matter can push a retrieval fetch past an answer engine's timeout. Rate-limit at the edge instead, where enforcement is real and applies uniformly.

Agents that are outside the contract by design

Several documented agents state that they may not follow robots.txt. Perplexity says Perplexity-User "generally ignores robots.txt rules". Meta says meta-externalfetcher "may bypass robots.txt rules", and facebookexternalhit may bypass it for "security or integrity checks". OpenAI's ChatGPT-User and Amazon's Amzn-User use softer wording: rules "may not apply" and "may not follow all robots.txt directives". These are all user-initiated fetchers, acting on behalf of a person who asked for a specific page. Writing a Disallow for them is a statement of preference, not a control. Our crawler registry records this per agent, so a report does not claim you have blocked something you have not.

Two tokens are the inverse case: Google-Extended and Applebot-Extended emit no user agent at all. Google states that "Google-Extended doesn't have a separate HTTP request user agent string" and that the token is used "in a control capacity"; Apple states that "Applebot-Extended does not crawl webpages". Your posture toward these two exists only in robots.txt, and no log analysis can tell you anything about them.

A worked reference file

This implements one common policy: allow answer-engine retrieval and classic search, deny model training. Adapt the token lists to your own decision rather than copying the policy.

# ---------------------------------------------------------------------------
# robots.txt for example.com   -   reviewed 2026-08-22
# Policy: allow retrieval and search, deny training.
# This file governs THIS HOST ONLY. docs.example.com needs its own copy.
# No Crawl-delay: support is split (Anthropic honours it, Apple and Amazon
# do not, Google's spec omits it). Rate limits belong at the edge.
# ---------------------------------------------------------------------------

# --- Answer-engine retrieval: allowed everywhere public --------------------
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: MistralAI-Index
User-agent: DuckAssistBot
User-agent: Amzn-SearchBot
Disallow: /cart/
Disallow: /account/
Disallow: /internal/
Disallow: /*?
Allow: /*?page=

# --- Classic search index, which is also the crawl behind AI Overviews ----
User-agent: Googlebot
User-agent: bingbot
User-agent: Applebot
User-agent: Amazonbot
Disallow: /cart/
Disallow: /account/
Disallow: /internal/
Disallow: /reports/
Allow: /reports/public/

# --- Training crawlers: denied --------------------------------------------
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

# --- Control tokens: no crawler, robots.txt is the only surface -----------
User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# --- Everything not named above -------------------------------------------
# Reached only by crawlers with no group of their own. It neither loosens
# nor tightens any group above: rules never cross group boundaries.
User-agent: *
Disallow: /cart/
Disallow: /account/
Disallow: /internal/

Sitemap: https://example.com/sitemap.xml

Three details in that file repay attention. Disallow: /*? followed by Allow: /*?page= works because the Allow pattern is longer, so paginated listings stay fetchable while every other query string is closed. Disallow: /reports/ with Allow: /reports/public/ is the same mechanism at directory level. And the * group repeats the shared disallows rather than relying on inheritance, because inheritance does not exist.

X-Robots-Tag, noindex and the snippet directives

robots.txt controls the fetch. The robots meta tag and the X-Robots-Tag response header control what happens to the page after it is fetched. They share one directive vocabulary, they combine, and the most restrictive value wins. The header form is the only option for non-HTML resources such as PDFs, images and JSON, and it can be set at the CDN without touching a template.

X-Robots-Tag: noindex, nofollow
X-Robots-Tag: max-snippet:120
X-Robots-Tag: applebot: nosnippet

<meta name="robots" content="max-snippet:120, noarchive">

<p>Public summary that may be quoted.
  <span data-nosnippet>Internal note, never quotable.</span>
</p>

What does not work is the non-standard family. Meta's crawler documentation is blunt about it: "we make it easy for site managers and content owners to indicate their preferences by using industry-standard practices like robots.txt rather than non-standard formats like NoAI tags." Treat noai and noimageai meta tags as decoration.

Note the asymmetry when you choose between the two layers. A robots.txt disallow is cheap, applies before the fetch, and is enforced by nothing but goodwill. A header directive requires the crawler to fetch the page first, so it costs you the request but controls the display. If your goal is "index me, do not quote me", the header is the only tool that expresses it.

Verify rather than assume

Our scanner parses robots.txt per RFC 9309 and evaluates every token in the registry against the same document, so the report tells you which crawlers a rule actually covers rather than which ones you intended. The full procedure, including how a group is selected and how a live block is distinguished from a robots.txt disallow, is documented on the methodology page.

Scan your site to see, per crawler, which group in your robots.txt is selected, which rule decides the verdict at your root path, whether that verdict came from a rule or from allow-by-default, and which X-Robots-Tag or meta directives your origin is attaching on top of it.