Blocklist · Huawei

Sites that block PanguBot

502 of the 29,991 domains we have measured, 2%, disallow pangubot at the site root in robots.txt. PanguBot collects content that may be used to train a model.

Every domain here has withheld its content from Huawei's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

502 of 29,991 measured domains (2%) disallow pangubot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block PanguBot", measured 2026-11-21. https://crawlcensus.com/blocked/pangubot

The list 502 domains

RankDomainTLDAI access score
26 amazon.com .com 40
62 msn.com .com 41
121 amazonvideo.com .com 55
190 cnn.com .com 50
260 launchpad.net .net 52
318 amazon.co.uk .uk 40
377 amazon.de .de 40
494 amazon.co.jp .jp 40
533 wired.com .com 81
571 uol.com.br .br 82
609 amazon.fr .fr 40
623 amazon.co.za .za 40
627 dailymail.co.uk .uk 61
628 nintendo.com .com 61
635 amazon.ca .ca 40
659 amazon.in .in 40
662 cnet.com .com 79
671 lemonde.fr .fr 81
681 dailymail.com .com 61
682 primevideo.com .com 71
706 amazon.it .it 40
725 amazon.es .es 40
726 ted.com .com 70
763 bild.de .de 81
812 amazon.com.au .au 40
833 amazon.com.br .br 40
942 repubblica.it .it 85
955 corriere.it .it 63
983 elmundo.es .es 84
1,019 theconversation.com .com 62
1,032 amazon.com.mx .mx 40
1,042 spiegel.de .de 64
1,056 marca.com .com 85
1,073 bfmtv.com .com 81
1,098 franceinfo.fr .fr 60
1,177 pimpbunny.com .com 56
1,189 aftonbladet.se .se 77
1,208 vg.no .no 79
1,223 furaffinity.net .net 39
1,232 pandora.com .com 53
1,300 f95zone.to .to 59
1,355 iqiyi.com .com 61
1,376 dagbladet.no .no 74
1,394 usnews.com .com 83
1,409 iltalehti.fi .fi 58
1,793 newyorker.com .com 73
1,831 zdnet.com .com 76
1,984 themoviedb.org .org 74
1,993 ign.com .com 62
2,006 amazon.eg .eg 40
2,010 mashable.com .com 61
2,325 nrk.no .no 69
2,346 slate.com .com 80
2,423 pcmag.com .com 63
2,691 newscientist.com .com 76
2,723 yumpu.com .com 30
2,786 chicagotribune.com .com 79
2,838 mastodon.social .social 49
2,899 vanityfair.it .it 77
2,917 faz.net .net 39
2,975 imageshack.us .us 71
2,989 vogue.com .com 82
3,179 crunchbase.com .com 71
3,198 metro.co.uk .uk 59
3,224 tagesschau.de .de 80
3,276 myanimelist.net .net 70
3,286 flipboard.com .com 64
3,342 lifehacker.com .com 65
3,368 nzherald.co.nz .nz 56
3,437 tagesspiegel.de .de 74
3,499 amazon.se .se 40
3,520 amazon.ae .ae 40
3,533 vanityfair.com .com 81
3,542 orf.at .at 69
3,587 br.de .de 86
3,595 politico.eu .eu 62
3,598 amazon.nl .nl 40
3,617 boston.com .com 87
3,633 ndr.de .de 79
3,771 yle.fi .fi 85
3,796 ardmediathek.de .de 73
3,800 ebsco.com .com 65
3,869 pcworld.com .com 84
3,966 bostonglobe.com .com 83
4,065 seekingalpha.com .com 91
4,161 kompas.com .com 77
4,333 derstandard.at .at 56
4,354 computerworld.com .com 83
4,407 20minutos.es .es 87
4,431 swr.de .de 76
4,484 mdr.de .de 76
4,493 wdr.de .de 86
4,512 amazon.com.tr .tr 40
4,579 srf.ch .ch 87
4,592 lastampa.it .it 81
4,987 dr.dk .dk 81
5,037 imageshack.com .com 71
5,068 gq.com .com 81
5,146 everydayhealth.com .com 60
5,158 mercurynews.com .com 78
5,272 depositphotos.com .com 63
5,310 kotaku.com .com 89
5,357 larazon.es .es 81
5,401 mediamarkt.de .de 90
5,523 architecturaldigest.com .com 83
5,547 amazon.ie .ie 40
5,678 dn.se .se 60
5,684 theepochtimes.com .com 80
5,714 francetvinfo.fr .fr 53
5,726 chefkoch.de .de 81
5,752 sankei.com .com 76
5,874 hs.fi .fi 68
5,879 cio.com .com 86
6,064 delfi.lt .lt 83
6,089 azcentral.com .com 74
6,125 denverpost.com .com 78
6,180 tf1info.fr .fr 84
6,246 eurogamer.net .net 57
6,352 tv2.dk .dk 74
6,356 freep.com .com 73
6,361 ici.fr .fr 87
6,512 telegram.com .com 73
6,513 sandiegouniontribune.com .com 78
6,593 wired.co.uk .uk 64
6,610 is.fi .fi 68
6,702 infoworld.com .com 83
6,793 svt.se .se 66
6,804 sportschau.de .de 85
6,808 chunichi.co.jp .jp 71
6,816 maxroll.gg .gg 59
6,895 togetter.com .com 73
6,948 nicematin.com .com 77
7,042 cntraveler.com .com 82
7,106 nrc.nl .nl 84
7,127 hani.co.kr .kr 67
7,178 sanspo.com .com 77
7,201 game8.jp .jp 72
7,239 gala.de .de 76
7,275 baltimoresun.com .com 76
7,372 15min.lt .lt 74
7,455 amazon.sg .sg 40
7,465 nationalgeographic.org .org 82
7,527 statnews.com .com 89
7,562 nettavisen.no .no 63
7,738 pitchfork.com .com 81
7,863 refinery29.com .com 78
8,094 nefisyemektarifleri.com .com 76
8,268 litmarket.ru .ru 87
8,380 veikkaus.fi .fi 67
8,450 complex.com .com 91
8,484 sciencenews.org .org 86
8,830 portalsamorzadowy.pl .pl 69
8,897 rynekzdrowia.pl .pl 64
8,959 glamour.com .com 81
9,069 wbur.org .org 81
9,149 forth.gr .gr 60
9,302 macworld.com .com 85
9,339 detroitnews.com .com 68
9,388 law.com .com 68
9,885 wral.com .com 87
10,279 klingai.com .com 78
10,291 usmagazine.com .com 87
10,345 inews.co.uk .uk 67
10,400 morgenpost.de .de 81
10,432 expansion.com .com 81
10,470 babycenter.com .com 62
10,501 ocregister.com .com 80
10,552 self.com .com 68
10,585 anyflip.com .com 56
10,708 vogue.co.uk .uk 79
10,826 bonappetit.com .com 82
10,885 livestrong.com .com 80
10,894 augsburger-allgemeine.de .de 81
10,933 rts.ch .ch 69
10,934 huffingtonpost.fr .fr 81
10,952 informationweek.com .com 82
11,200 aftenposten.no .no 76
11,217 businessinsider.de .de 76
11,242 teenvogue.com .com 82
11,349 posteo.de .de 67
11,405 svd.se .se 78
11,457 wnp.pl .pl 65
11,516 cults3d.com .com 91
11,610 allure.com .com 84
11,659 abendblatt.de .de 65
12,113 bostonherald.com .com 78
12,294 ally.com .com 94
12,377 delfi.ee .ee 57
12,533 indystar.com .com 73
12,542 epicurious.com .com 64
12,743 extremetech.com .com 64
12,972 noticiasaominuto.com .com 85
12,990 nybooks.com .com 80
13,169 csoonline.com .com 86
13,201 hessenschau.de .de 87
13,318 contentmarketinginstitute.com .com 82
13,398 gq-magazine.co.uk .uk 81
13,579 barcelona.cat .cat 82
13,667 dispatch.com .com 74
13,772 khan.co.kr .kr 72
Page 1 of 3Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the PanguBot reference, the full census, and the JSON endpoint.