Sites that block Googlebot
17 of the 29,991 domains we have measured, 0.1%, disallow googlebot at the site root in robots.txt. Googlebot builds the index that decides whether the operator's assistant cites a page.
Every domain here has removed itself from Google's answers. Blocking a retrieval agent withholds nothing from model training, because training uses a different token.
17 of 29,991 measured domains (0.1%) disallow googlebot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.
Measured 2026-11-22 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.
Cite as: Crawl Census, "Sites that block Googlebot", measured 2026-11-22. https://crawlcensus.com/blocked/googlebot
The list 17 domains
| Rank | Domain | TLD | AI access score |
|---|---|---|---|
| 1,754 | ad-contents.jp | .jp | 45 |
| 4,200 | disdarrummers.shop | .shop | 82 |
| 4,619 | dextrodedenda.top | .top | 82 |
| 5,597 | feinidi.com | .com | 82 |
| 7,228 | ishe88.com | .com | 82 |
| 10,172 | dyz9.top | .top | 82 |
| 11,468 | nizixun.com | .com | 82 |
| 16,122 | page-stats.de | .de | 49 |
| 20,137 | 3bb.co.th | .th | 71 |
| 23,458 | inveragil.com | .com | 34 |
| 25,270 | startersites.io | .io | 46 |
| 25,826 | cnnic.cn | .cn | 58 |
| 25,857 | ly.com | .com | 69 |
| 29,683 | officeplus.cn | .cn | 60 |
| 30,868 | sosalkino.best | .best | 69 |
| 39,484 | brighteon.com | .com | 56 |
| 44,129 | eeo.cn | .cn | 54 |
Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.
Check a specific domain
See also the Googlebot reference, the full census, and the JSON endpoint.