Crawler operators
Who runs the crawlers
48 tracked agents belong to 23 operators. Each page collects everything measured about that company's crawling, and carries a correction channel for the company itself.
| Operator | Agents | Documented as ignoring robots.txt |
|---|---|---|
| Meta | 6 | 2 |
| 5 | - | |
| OpenAI | 4 | - |
| Anthropic | 4 | - |
| Amazon | 3 | 1 |
| Mistral AI | 3 | - |
| Apple | 2 | - |
| Microsoft | 2 | - |
| Perplexity | 2 | 1 |
| ByteDance | 2 | 2 |
| Webz.io | 2 | - |
| Cohere | 2 | - |
| Common Crawl Foundation | 1 | - |
| Diffbot | 1 | - |
| Allen Institute for AI | 1 | - |
| DuckDuckGo | 1 | - |
| You.com | 1 | - |
| Huawei | 1 | - |
| Timpi | 1 | - |
| ImageSift (Hive) | 1 | - |
| Kangaroo LLM | 1 | - |
| Semrush | 1 | - |
| Zyte (open-source framework) | 1 | - |