Let ChatGPT cite you while blocking AI training

The answer

Every major operator has now split its robots.txt tokens by purpose, so "allow citation, deny training" is a file you can write rather than a preference you can only hope for. Allow the retrieval and search tokens. Disallow the training tokens. Write both as explicit groups, because a crawler with its own group never reads anyone else's rules.

The split is not symmetrical across vendors, and three of the tokens involved crawl nothing at all. Get the mapping right first, then copy the file.

The token split, vendor by vendor

Sources, all read 2026-08-21: OpenAI crawler documentation, Anthropic crawler support article, Perplexity crawler documentation, Mistral robots documentation, Amazonbot documentation, Google crawler documentation, Applebot documentation, Meta web crawlers documentation, DuckAssistBot help page. Full per-crawler detail is on the crawler registry.

The file

This is complete and copy-pasteable. Replace the sitemap hostname and nothing else.

# Retrieval, search indexing, and answer-engine citation: allowed.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: DuckAssistBot
User-agent: meta-webindexer
User-agent: Applebot
User-agent: Googlebot
User-agent: bingbot
Allow: /

# Foundation-model training: denied.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: MistralAI-Training
User-agent: Amazonbot
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-training-data-crawler
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including crawlers that do not exist yet.
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

OpenAI states the trade-off directly

"Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

That single sentence from the OpenAI crawler documentation (read 2026-08-21) is the clearest statement any vendor makes about the cost of over-blocking, and it is why a blanket User-agent: * plus Disallow: / is a different decision from an anti-training policy. OpenAI's own recommendation is to allow the search token and its published IP ranges. The same page notes that ChatGPT-User "is not used to determine whether content may appear in Search", so blocking the user-initiated fetcher does not protect you from anything and blocking it does not remove you from search either. If both are allowed, OpenAI says it may reuse one crawl for both purposes to avoid duplicate fetches.

No other operator publishes an equally explicit consequence, but the structure is identical everywhere: the retrieval token is the one that decides whether you can be cited, and it is a different token from the trainer. Denying GPTBot while allowing OAI-SearchBot is a coherent position. Denying both is a decision to be absent.

Nothing you change takes effect today

Vendor-published figures, from the operator documentation linked above, read 2026-08-21.

Amazon's is the one that bites. A month-long cache means a fix you deployed this morning is not the policy Amazonbot is operating under, and the Amazonbot documentation adds a second trap: if Amazon cannot fetch your robots.txt, it "will behave as if it does not exist". A file that 500s, times out, or sits behind a challenge page is not a strict policy, it is no policy. The same is true of any 4xx that your application router turns into a styled HTML error page at /robots.txt.

Two fallbacks that overrule your file

Both are documented, both are easy to miss, and both mean the effective rule for an agent is not readable from its own group.

• Applebot follows Googlebot rules when it is not mentioned. Apple states: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions" (Applebot documentation, read 2026-08-21). A site that blocks Googlebot from a section and never names Applebot has blocked Apple Intelligence from that section by accident. The inverse is the more common case: sites that assume Apple is excluded because they never mentioned it are in fact governed by whatever Googlebot is allowed.
• Amzn-SearchBot falls back to other search-bot rules the same way (Amazonbot documentation). If your file is generous to Googlebot and bingbot and silent about Amazon's search fetcher, that fetcher inherits the generous rules.

Practical consequence: name every agent you have an opinion about. Silence is not neutrality for these two.

The agents that are outside the contract entirely

Four documented agents tell you in advance that a Disallow may not stop them. Reporting them as "blocked" would overstate what robots.txt controls, so we do not.

• Perplexity-User "generally ignores robots.txt rules" (Perplexity crawler documentation).
• meta-externalfetcher "may bypass robots.txt rules", and facebookexternalhit may bypass for security or integrity checks (Meta web crawlers documentation).
• ChatGPT-User: "robots.txt rules may not apply" (OpenAI crawler documentation).
• Amzn-User "may not follow all robots.txt directives" (Amazonbot documentation).

The honest framing is that user-initiated fetchers sit outside the robots.txt contract by design: a person asked an assistant to open your page, and the vendors treat that as closer to a browser than to a crawl. This is also why the file above allows them. Allowing an agent that would have come anyway costs nothing and keeps your declared policy consistent with observable behaviour.

Verify with IP ranges, not user agents

A user-agent string is a claim, not an identity. Common Crawl says plainly that it is "aware of crawlers falsely identifying themselves as CCBot". Every operator worth allowing publishes a machine-readable IP list, and matching against it is the only way to know a request was really theirs: openai.com/searchbot.json, claude.com/crawling/bots.json, perplexity.com/perplexitybot.json, mistral.ai/mistralai-index-ips.json, developer.amazon.com/amazonbot/searchbot-ip-addresses/, search.developer.apple.com/applebot.json and Google's common-crawlers.json. Google and Apple also support reverse DNS, under crawl-*.googlebot.com and *.applebot.apple.com.

Two more mechanics worth knowing while you are in the file. robots.txt is fetched per host, so docs.example.com/robots.txt governs that hostname independently of the apex and has to be audited separately. And Crawl-delay is not portable: Anthropic honors it, Apple and Amazon explicitly do not, and it is not part of Google's specification.

Where robots.txt stops being the answer

A permissive file means nothing if your edge returns a challenge to anything whose user agent contains "bot". That is the single most common cause of a site that believes it is open and is not, and it is invisible from the file alone. It is also worth remembering that Google-Extended and Applebot-Extended send no user agent at all: they exist only as control tokens, so no log analysis can tell you your posture toward them. robots.txt is the only place that answer lives.

A Crawl Census scan reads your file the way RFC 9309 says a crawler must, evaluates every registry token against it, then re-requests your home page under four real crawler user agents to see whether the edge agrees with the file. Run a scan to get the per-token verdict for all six vendor splits above, plus a flag on every agent your firewall is refusing regardless of what robots.txt says.