Sites that block msnbot
18 of the 29,991 domains we have measured, 0.1%, disallow msnbot at the site root in robots.txt. msnbot builds the index that decides whether the operator's assistant cites a page.
Every domain here has removed itself from Microsoft's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.
18 of 29,991 measured domains (0.1%) disallow msnbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.
Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "Sites that block msnbot", measured 2026-11-22. https://crawlcensus.com/blocked/msnbot
The list 18 domains
| Rank | Domain | TLD | AI access score |
|---|---|---|---|
| 3,872 | cia.gov | .gov | 49 |
| 5,840 | uzh.ch | .ch | 79 |
| 14,805 | redalyc.org | .org | 52 |
| 16,122 | page-stats.de | .de | 49 |
| 16,556 | tourvisor.ru | .ru | 91 |
| 21,162 | psytests.org | .org | 69 |
| 22,908 | redvelvet.co.za | .za | 73 |
| 24,525 | zin.ru | .ru | 41 |
| 25,638 | truenetwork.ru | .ru | 70 |
| 25,826 | cnnic.cn | .cn | 58 |
| 25,857 | ly.com | .com | 69 |
| 28,163 | segmento.ru | .ru | 70 |
| 28,335 | giant-bicycles.com | .com | 89 |
| 29,121 | pucsp.br | .br | 55 |
| 30,393 | ecode360.com | .com | 44 |
| 34,818 | securitm.ru | .ru | 92 |
| 37,425 | jamestown.org | .org | 82 |
| 43,837 | primicias.ec | .ec | 93 |
Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.
Check a specific domain
See also the msnbot reference, the full census, and the JSON endpoint.