Blocklist · Huawei

Sites that block PanguBot

502 of the 29,991 domains we have measured, 2%, disallow pangubot at the site root in robots.txt. PanguBot collects content that may be used to train a model.

Every domain here has withheld its content from Huawei's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

502 of 29,991 measured domains (2%) disallow pangubot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block PanguBot", measured 2026-11-21. https://crawlcensus.com/blocked/pangubot

The list 502 domains

RankDomainTLDAI access score
40,796 elpasotimes.com .com 73
40,874 grid.id .id 64
40,876 cntraveller.in .in 81
40,969 nrn.com .com 81
41,098 gqmagazine.fr .fr 82
41,121 lancashiretelegraph.co.uk .uk 78
41,336 archinect.com .com 68
41,349 planet-wissen.de .de 71
41,394 capitalizemytitle.com .com 71
41,519 greenme.it .it 82
41,542 oefb.at .at 50
41,603 lci.fr .fr 84
41,647 cjonline.com .com 73
41,736 eadt.co.uk .uk 79
41,760 dailybreeze.com .com 76
41,826 defcon.org .org 66
41,872 vanityfair.fr .fr 66
41,947 vogue.co.jp .jp 65
41,958 theledger.com .com 74
42,054 lecker.de .de 88
42,076 iex.nl .nl 48
42,113 glasgowtimes.co.uk .uk 79
42,160 tcpalm.com .com 73
42,614 citizen-times.com .com 74
42,651 computerwoche.de .de 85
42,698 sol.no .no 74
43,132 jtbc.co.kr .kr 70
43,207 tv4play.se .se 77
43,302 seacoastonline.com .com 69
43,548 tvmovie.de .de 89
43,587 admagazine.fr .fr 81
43,593 mediamarkt.at .at 89
43,605 zonerama.com .com 30
43,986 nn.de .de 83
43,996 supermarketnews.com .com 82
44,026 gainesville.com .com 66
44,252 braunschweiger-zeitung.de .de 65
44,807 gofeminin.de .de 74
44,874 news-leader.com .com 62
44,878 rtvnoord.nl .nl 73
44,908 mora.jp .jp 45
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,159 lansingstatejournal.com .com 62
45,240 vogue.mx .mx 65
45,272 rtvutrecht.nl .nl 73
45,316 eumetsat.int .int 80
45,349 wp.de .de 65
45,440 nrz.de .de 80
45,578 chugoku-np.co.jp .jp 80
45,604 beaconjournal.com .com 66
45,680 greenvilleonline.com .com 62
45,716 ithome.com.tw .tw 56
45,762 autozeitung.de .de 86
45,846 amazon.jp .jp 40
45,905 kob.com .com 84
45,918 revistagq.com .com 81
45,989 aftenbladet.no .no 78
46,231 registerguard.com .com 66
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,805 omroepwest.nl .nl 73
46,923 alisweb.org .org 39
46,949 schulferien.org .org 51
47,029 lostarmour.info .info 51
47,173 eurogamer.de .de 57
47,285 bengo4.com .com 86
47,299 rtvdrenthe.nl .nl 73
47,428 wealthmanagement.com .com 81
47,526 allgaeuer-zeitung.de .de 81
47,624 lacucinaitaliana.it .it 81
47,659 nationalgeographic.de .de 82
47,741 salzburg24.at .at 85
47,796 topekalibraries.info .info 46
47,884 vcstar.com .com 62
48,444 vogue.com.tw .tw 81
48,550 argusleader.com .com 62
48,777 mtv.fi .fi 76
48,858 otz.de .de 65
48,859 castro.fm .fm 67
49,061 posfie.com .com 73
49,067 gqitalia.it .it 64
49,194 mailonsunday.co.uk .uk 61
49,271 eltern.de .de 72
49,279 presstelegram.com .com 80
49,420 gds.it .it 77
49,447 businessinsider.es .es 87
49,501 sj-r.com .com 64
49,574 montgomeryadvertiser.com .com 62
49,802 pazar3.mk .mk 73
49,895 butenunbinnen.de .de 81
49,925 aamulehti.fi .fi 66
50,185 miricanvas.com .com 89
50,278 burlingtonfreepress.com .com 62
50,338 ohtuleht.ee .ee 65
50,367 blipfoto.com .com 57
50,394 ardaudiothek.de .de 85
50,520 womansworld.com .com 85
50,684 gq.com.mx .mx 72
50,933 nwaonline.com .com 73
50,960 hot.com .com 67
50,982 nrtool.st .st 54
PreviousPage 3 of 3

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the PanguBot reference, the full census, and the JSON endpoint.