Weekly report · 2026-W34
The state of AI crawling
Measured on 30,218 reachable domains for the week ending 2026-08-23. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
30,218
domains measured
7%
block an answer engine
19%
refuse AI agents at the edge
14%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 11.4% | 3,139 of 27,482 | new |
| CCBot | Common Crawl Foundation | archive | 11.2% | 3,089 of 27,482 | new |
| Bytespider | ByteDance | training | 10.8% | 2,959 of 27,482 | new |
| ClaudeBot | Anthropic | training | 10.0% | 2,747 of 27,482 | new |
| Google-Extended | training | 9.1% | 2,488 of 27,482 | new | |
| Meta-ExternalAgent | Meta | training | 9.0% | 2,478 of 27,482 | new |
| Amazonbot | Amazon | training | 9.0% | 2,460 of 27,482 | new |
| Applebot-Extended | Apple | training | 8.8% | 2,412 of 27,482 | new |
| anthropic-ai | Anthropic | training | 4.2% | 1,154 of 27,482 | new |
| omgilibot | Webz.io | training | 4.2% | 1,145 of 27,482 | new |
| Diffbot | Diffbot | training | 3.8% | 1,057 of 27,482 | new |
| cohere-ai | Cohere | live retrieval | 3.8% | 1,045 of 27,482 | new |
| omgili | Webz.io | training | 3.8% | 1,043 of 27,482 | new |
| SemrushBot-OCOB | Semrush | training | 3.7% | 1,012 of 27,482 | new |
| PerplexityBot | Perplexity | answer index | 3.6% | 1,002 of 27,482 | new |
| ChatGPT-User | OpenAI | live retrieval | 3.6% | 993 of 27,482 | new |
| FacebookBot | Meta | training | 3.3% | 898 of 27,482 | new |
| YouBot | You.com | answer index | 2.8% | 778 of 27,482 | new |
| ImagesiftBot | ImageSift (Hive) | training | 2.8% | 758 of 27,482 | new |
| Timpibot | Timpi | training | 2.7% | 755 of 27,482 | new |
| Scrapy | Zyte (open-source framework) | training | 2.3% | 640 of 27,482 | new |
| OAI-SearchBot | OpenAI | answer index | 2.2% | 614 of 27,482 | new |
| AI2Bot | Allen Institute for AI | training | 2.2% | 610 of 27,482 | new |
| cohere-training-data-crawler | Cohere | training | 2.0% | 540 of 27,482 | new |
| DuckAssistBot | DuckDuckGo | live retrieval | 1.7% | 480 of 27,482 | new |
| Meta-ExternalFetcher | Meta | live retrieval | 1.7% | 473 of 27,482 | new |
| PanguBot | Huawei | training | 1.7% | 462 of 27,482 | new |
| Claude-SearchBot | Anthropic | answer index | 1.6% | 435 of 27,482 | new |
| Claude-User | Anthropic | live retrieval | 1.5% | 424 of 27,482 | new |
| Kangaroo Bot | Kangaroo LLM | training | 1.5% | 417 of 27,482 | new |
| Perplexity-User | Perplexity | live retrieval | 1.5% | 411 of 27,482 | new |
| MistralAI-User | Mistral AI | live retrieval | 1.4% | 377 of 27,482 | new |
| Applebot | Apple | answer index | 1.2% | 326 of 27,482 | new |
| Google-CloudVertexBot | live retrieval | 1.2% | 323 of 27,482 | new | |
| Meta-WebIndexer | Meta | answer index | 1.1% | 302 of 27,482 | new |
| GoogleOther | training | 1.1% | 290 of 27,482 | new | |
| TikTokSpider | ByteDance | live retrieval | 0.8% | 218 of 27,482 | new |
| Amzn-SearchBot | Amazon | answer index | 0.7% | 183 of 27,482 | new |
| Amzn-User | Amazon | live retrieval | 0.4% | 104 of 27,482 | new |
| MistralAI-Index | Mistral AI | answer index | 0.2% | 65 of 27,482 | new |
| facebookexternalhit | Meta | live retrieval | 0.2% | 52 of 27,482 | new |
| MistralAI-Training | Mistral AI | training | 0.2% | 52 of 27,482 | new |
| Bingbot | Microsoft | answer index | 0.1% | 40 of 27,482 | new |
| Googlebot-News | answer index | 0.1% | 35 of 27,482 | new | |
| msnbot | Microsoft | answer index | 0.1% | 26 of 27,482 | new |
| Googlebot | answer index | 0.1% | 16 of 27,482 | new | |
| Meta-ExternalAds | Meta | training | 0.1% | 16 of 27,482 | new |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 1 of 27,482 | new |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 200 total
- amny.com GPTBot is now answered with HTTP 402
- amny.com OAI-SearchBot is no longer charged
- amny.com ClaudeBot is no longer charged
- wikipedia.org AI access score moved 74 to 64
- bhg.com AI access score moved 78 to 44
- petlove.com.br Published an llms.txt file
- abcactionnews.com GPTBot is now blocked by robots.txt
- abcactionnews.com Google-Extended is now blocked by robots.txt
- plainproxies.com Bytespider is now blocked by robots.txt
- plainproxies.com CCBot is now blocked by robots.txt
- plainproxies.com SemrushBot-OCOB is now blocked by robots.txt
- luizalabs.com GPTBot is now blocked by robots.txt
- luizalabs.com Google-Extended is now blocked by robots.txt
- spiritfanfiction.com GPTBot is now allowed by robots.txt
- spiritfanfiction.com ClaudeBot is now allowed by robots.txt
- spiritfanfiction.com PerplexityBot is now allowed by robots.txt
- spiritfanfiction.com Amazonbot is now allowed by robots.txt
- tor2me.info Published an llms.txt file
- aplaceformom.com GPTBot is now blocked by robots.txt
- aplaceformom.com ClaudeBot is now blocked by robots.txt
- aplaceformom.com CCBot is now blocked by robots.txt
- ujaen.es AI access score moved 69 to 50
- leetify.com GPTBot is now blocked by robots.txt
- leetify.com ChatGPT-User is now blocked by robots.txt
- leetify.com Google-Extended is now blocked by robots.txt
- leetify.com FacebookBot is now blocked by robots.txt
- leetify.com omgilibot is now blocked by robots.txt
- 1domainregistry.com Published an llms.txt file
- radiozet.pl GPTBot is now blocked by robots.txt
- radiozet.pl anthropic-ai is now blocked by robots.txt
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2026-W34", 2026-08-23. https://crawlcensus.com/report/2026-W34Machine-readable: census totals, full corpus, crawler registry. Method: methodology.