Crawl Census
Scan a site
Blocklist · Microsoft

Sites that block msnbot

26 of the 25,096 domains we have measured, 0.1%, disallow msnbot at the site root in robots.txt. msnbot builds the index that decides whether the operator's assistant cites a page.

Every domain here has removed itself from Microsoft's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.

The measurement

26 of 25,096 measured domains (0.1%) disallow msnbot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-08-23 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block msnbot", measured 2026-08-23. https://crawlcensus.com/blocked/msnbot

The list 26 domains

RankDomainTLDAI access score
1,988 mos.ru .ru 74
3,872 cia.gov .gov 46
5,371 rutracker.org .org 46
5,840 uzh.ch .ch 77
9,576 kgirls.net .net 62
14,805 redalyc.org .org 52
16,122 page-stats.de .de 49
16,556 tourvisor.ru .ru 91
21,162 psytests.org .org 69
21,393 desifakes.com .com 58
21,930 endnote.com .com 85
22,908 redvelvet.co.za .za 73
23,342 enrollbusiness.com .com 36
24,525 zin.ru .ru 54
25,252 goszakup.gov.kz .kz 73
25,826 cnnic.cn .cn 59
25,857 ly.com .com 68
28,163 segmento.ru .ru 70
28,335 giant-bicycles.com .com 85
29,121 pucsp.br .br 55
30,393 ecode360.com .com 48
33,108 tender.pro .pro 66
34,818 securitm.ru .ru 70
35,155 megahost.net .net 86
37,425 jamestown.org .org 84
40,753 decitre.fr .fr 66

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the msnbot reference, the full census, and the JSON endpoint.