Weekly report · 2011-W25
The state of AI crawling
Measured on 29,994 reachable domains for the week ending 2011-06-26. Every figure is the result of an automated audit run the same way on every domain: read robots.txt, resolve the rules for 48 tracked agents, then send live requests as four of them.
29,994
domains measured
7%
block an answer engine
19%
refuse AI agents at the edge
14%
publish an llms.txt
Block rate by crawler with change on the previous week
| Crawler | Operator | Content used for | Block rate | Sites | Week on week |
|---|---|---|---|---|---|
| GPTBot | OpenAI | training | 7.0% | 2,085 of 29,989 | new |
| CCBot | Common Crawl Foundation | archive | 6.8% | 2,037 of 29,989 | new |
| Bytespider | ByteDance | training | 6.4% | 1,912 of 29,989 | new |
| ClaudeBot | Anthropic | training | 5.7% | 1,695 of 29,989 | new |
| Meta-ExternalAgent | Meta | training | 4.6% | 1,375 of 29,989 | new |
| Amazonbot | Amazon | training | 4.5% | 1,364 of 29,989 | new |
| Google-Extended | training | 4.5% | 1,357 of 29,989 | new | |
| Applebot-Extended | Apple | training | 4.4% | 1,327 of 29,989 | new |
| anthropic-ai | Anthropic | training | 4.3% | 1,296 of 29,989 | new |
| omgilibot | Webz.io | training | 4.2% | 1,273 of 29,989 | new |
| cohere-ai | Cohere | live retrieval | 4.0% | 1,192 of 29,989 | new |
| Diffbot | Diffbot | training | 4.0% | 1,188 of 29,989 | new |
| omgili | Webz.io | training | 3.9% | 1,159 of 29,989 | new |
| SemrushBot-OCOB | Semrush | training | 3.6% | 1,091 of 29,989 | new |
| PerplexityBot | Perplexity | answer index | 3.5% | 1,042 of 29,989 | new |
| ChatGPT-User | OpenAI | live retrieval | 3.5% | 1,038 of 29,989 | new |
| FacebookBot | Meta | training | 3.2% | 961 of 29,989 | new |
| YouBot | You.com | answer index | 2.9% | 876 of 29,989 | new |
| ImagesiftBot | ImageSift (Hive) | training | 2.8% | 835 of 29,989 | new |
| Timpibot | Timpi | training | 2.7% | 813 of 29,989 | new |
| Scrapy | Zyte (open-source framework) | training | 2.4% | 728 of 29,989 | new |
| AI2Bot | Allen Institute for AI | training | 2.3% | 678 of 29,989 | new |
| OAI-SearchBot | OpenAI | answer index | 2.1% | 628 of 29,989 | new |
| cohere-training-data-crawler | Cohere | training | 2.0% | 600 of 29,989 | new |
| Meta-ExternalFetcher | Meta | live retrieval | 1.8% | 535 of 29,989 | new |
| PanguBot | Huawei | training | 1.7% | 499 of 29,989 | new |
| DuckAssistBot | DuckDuckGo | live retrieval | 1.7% | 497 of 29,989 | new |
| Claude-User | Anthropic | live retrieval | 1.6% | 473 of 29,989 | new |
| Claude-SearchBot | Anthropic | answer index | 1.5% | 461 of 29,989 | new |
| Kangaroo Bot | Kangaroo LLM | training | 1.5% | 451 of 29,989 | new |
| Perplexity-User | Perplexity | live retrieval | 1.5% | 443 of 29,989 | new |
| MistralAI-User | Mistral AI | live retrieval | 1.4% | 429 of 29,989 | new |
| Google-CloudVertexBot | live retrieval | 1.3% | 395 of 29,989 | new | |
| GoogleOther | training | 1.2% | 354 of 29,989 | new | |
| Meta-WebIndexer | Meta | answer index | 1.2% | 345 of 29,989 | new |
| Applebot | Apple | answer index | 1.1% | 331 of 29,989 | new |
| TikTokSpider | ByteDance | live retrieval | 0.8% | 246 of 29,989 | new |
| Amzn-SearchBot | Amazon | answer index | 0.7% | 215 of 29,989 | new |
| Amzn-User | Amazon | live retrieval | 0.5% | 148 of 29,989 | new |
| MistralAI-Training | Mistral AI | training | 0.4% | 118 of 29,989 | new |
| MistralAI-Index | Mistral AI | answer index | 0.2% | 73 of 29,989 | new |
| facebookexternalhit | Meta | live retrieval | 0.2% | 52 of 29,989 | new |
| Bingbot | Microsoft | answer index | 0.1% | 39 of 29,989 | new |
| Googlebot-News | answer index | 0.1% | 38 of 29,989 | new | |
| Meta-ExternalAds | Meta | training | 0.1% | 22 of 29,989 | new |
| msnbot | Microsoft | answer index | 0.1% | 22 of 29,989 | new |
| Googlebot | answer index | 0.1% | 18 of 29,989 | new | |
| OAI-AdsBot | OpenAI | live retrieval | 0.0% | 9 of 29,989 | new |
Trend share of measured sites
Where the blocking is share blocking an answer engine, by top-level domain
Only TLDs with at least 40 measured domains are shown, and each bar carries its sample size. A TLD is a weak proxy for jurisdiction. Full segmentation.
Changes recorded this week 0 total
No crawler policy changes were recorded in this week.
Cite this CC BY 4.0
The figures on this page are frozen to the week shown and will not be edited after publication. Corrections are appended, not overwritten.
Crawl Census, "The state of AI crawling, 2011-W25", 2011-06-26. https://crawlcensus.com/report/2011-W25Machine-readable: census totals, full corpus, crawler registry. Method: methodology.