Blocklist · Huawei

Sites that block PanguBot

502 of the 29,991 domains we have measured, 2%, disallow pangubot at the site root in robots.txt. PanguBot collects content that may be used to train a model.

Every domain here has withheld its content from Huawei's training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

502 of 29,991 measured domains (2%) disallow pangubot at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block PanguBot", measured 2026-11-21. https://crawlcensus.com/blocked/pangubot

The list 502 domains

RankDomainTLDAI access score
13,825 heraldscotland.com .com 81
13,860 francebleu.fr .fr 87
13,871 ilex.by .by 59
13,879 tennessean.com .com 74
13,882 vermont.gov .gov 73
14,090 simon.com .com 31
14,374 delfi.lv .lv 80
14,470 gamedeveloper.com .com 83
14,709 dazeddigital.com .com 78
14,733 waz.de .de 82
14,826 rockpapershotgun.com .com 58
15,174 pcgamesn.com .com 83
15,218 statesman.com .com 51
15,269 gamesindustry.biz .biz 57
15,321 coursehero.com .com 76
15,372 letudiant.fr .fr 81
15,401 desmoinesregister.com .com 74
15,458 utopia.de .de 85
15,476 computerbild.de .de 81
15,557 cincinnati.com .com 74
15,686 postimees.ee .ee 66
16,082 t3n.de .de 78
16,086 nintendo.co.jp .jp 50
16,252 vivino.com .com 82
16,254 unil.ch .ch 83
16,290 duden.de .de 71
16,323 ard.de .de 76
16,428 e24.no .no 78
16,564 mapgenie.io .io 59
16,664 lightningmaps.org .org 51
16,795 vozpopuli.com .com 87
16,814 whattoexpect.com .com 65
17,042 northjersey.com .com 74
17,183 semafor.com .com 72
17,455 twincities.com .com 81
17,469 palmbeachpost.com .com 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,694 manager-magazin.de .de 63
17,886 agrarheute.com .com 75
18,280 hoerzu.de .de 45
18,308 preyproject.com .com 93
18,442 empireonline.com .com 62
18,489 elcolombiano.com .com 83
18,549 gamasutra.com .com 83
18,863 dailynews.com .com 76
18,870 scotusblog.com .com 83
18,883 courier-journal.com .com 74
18,894 courrierinternational.com .com 83
19,250 beatstars.com .com 51
19,690 equidia.fr .fr 65
19,860 zawya.com .com 80
20,318 supersport.hr .hr 51
20,364 derwesten.de .de 83
20,419 01net.com .com 93
20,761 nlc.hu .hu 85
20,958 pressdemocrat.com .com 80
21,004 femina.hu .hu 82
21,038 mainpost.de .de 80
21,231 epochtimes.com .com 85
21,267 seher.no .no 70
21,823 desy.de .de 62
21,889 scoop.co.nz .nz 67
22,007 vg247.com .com 58
22,623 mcall.com .com 78
23,084 app.com .com 69
23,154 rbb24.de .de 80
23,613 nordbayern.de .de 81
23,964 pcwelt.de .de 78
24,014 askmen.com .com 47
24,102 knoxnews.com .com 74
24,141 cntraveller.com .com 81
24,190 jacksonville.com .com 74
24,233 glamourmagazine.co.uk .uk 81
24,525 zin.ru .ru 41
24,593 vogue.in .in 64
24,714 occrp.org .org 62
24,882 uol.com .com 82
24,934 svtplay.se .se 78
24,946 vogue.fr .fr 81
24,977 wired.it .it 83
24,983 sn.at .at 70
25,031 franceculture.fr .fr 81
25,048 tokyo-np.co.jp .jp 81
25,115 pilotonline.com .com 79
25,478 franceinter.fr .fr 80
25,776 zinio.com .com 76
25,839 delawareonline.com .com 74
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
26,114 democratandchronicle.com .com 73
26,210 kstp.com .com 83
26,474 mbta.com .com 64
26,637 brigitte.de .de 79
26,806 golf.com .com 88
26,853 tallahassee.com .com 74
26,963 premiumoutlets.com .com 31
27,078 mdzol.com .com 61
27,182 ultimahora.es .es 83
27,256 amny.com .com 83
27,401 derstandard.de .de 56
27,723 geek.com .com 38
27,756 beatstars.net .net 51
27,838 mtvuutiset.fi .fi 72
27,999 earth.com .com 80
28,170 suedkurier.de .de 81
28,383 lightreading.com .com 81
28,573 wpr.org .org 88
28,616 amzn.com .com 40
28,642 nachrichten.at .at 82
28,678 thenational.scot .scot 81
28,775 vogue.de .de 65
28,792 uesp.net .net 55
28,794 eastbaytimes.com .com 79
29,048 gq-magazin.de .de 66
29,115 gp.se .se 78
29,139 u-cursos.cl .cl 51
29,323 goop.com .com 66
29,390 kompas.id .id 64
29,499 conjur.com.br .br 73
29,553 hankookilbo.com .com 84
29,692 piliapp.com .com 65
29,821 providencejournal.com .com 73
30,004 sfstandard.com .com 72
30,074 yugipedia.com .com 58
30,237 clarionledger.com .com 74
30,387 bsc.es .es 37
30,443 dobrzemieszkaj.pl .pl 61
30,703 wickedlocal.com .com 75
30,810 kentonline.co.uk .uk 76
30,974 techadvisor.com .com 85
30,986 commercialappeal.com .com 74
30,992 huffingtonpost.it .it 83
31,032 howlongtobeat.com .com 49
31,176 edp24.co.uk .uk 79
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,782 mastodon.online .online 49
31,786 thespec.com .com 64
32,169 journaldugeek.com .com 91
32,267 theargus.co.uk .uk 79
32,388 rgj.com .com 73
32,437 hoopshype.com .com 70
32,460 saturn.de .de 91
32,713 opovo.com.br .br 85
32,718 athletic.net .net 68
32,735 rsi.ch .ch 63
32,790 tatler.com .com 82
32,803 oxfordmail.co.uk .uk 79
32,806 tvnet.lv .lv 79
33,430 ilmanifesto.it .it 68
33,494 theanalyst.com .com 88
33,562 myporntube.com .com 86
33,644 tv4.se .se 76
33,738 ruv.is .is 59
34,097 vogue.es .es 65
34,309 unionleader.com .com 63
34,445 monoskop.org .org 43
34,750 mailplus.co.uk .uk 44
34,973 naplesnews.com .com 73
34,986 ruhrnachrichten.de .de 75
35,062 dailypress.com .com 79
35,326 swr3.de .de 74
35,364 road.cc .cc 84
35,517 datacenterknowledge.com .com 82
35,543 news-journalonline.com .com 73
35,759 houseandgarden.co.uk .uk 81
35,763 penguin.de .de 88
35,803 housepricesintheuk.co.uk .uk 74
35,955 oekotest.de .de 86
36,334 ytmnd.com .com 41
36,356 elgaronline.com .com 45
36,539 nd-aktuell.de .de 78
36,575 dailyecho.co.uk .uk 81
37,297 storyblocks.com .com 70
37,547 bt.no .no 77
38,007 wired.jp .jp 78
38,129 11freunde.de .de 58
38,148 news-press.com .com 73
38,193 ardsounds.de .de 85
38,327 inforum.com .com 70
38,409 igt.com .com 48
38,507 bournemouthecho.co.uk .uk 79
38,519 kochbar.de .de 62
38,668 stimme.de .de 72
38,739 thueringer-allgemeine.de .de 82
39,206 futurezone.de .de 86
39,393 zakzak.co.jp .jp 72
39,657 korben.info .info 77
39,824 werstreamt.es .es 59
40,184 estantevirtual.com.br .br 91
40,317 iza.ne.jp .jp 79
40,328 bntu.by .by 80
40,339 kompas.tv .tv 76
40,512 thetelegraphandargus.co.uk .uk 78
40,638 main-echo.de .de 64
40,713 dwds.de .de 64
40,728 mediamarkt.pl .pl 89
PreviousPage 2 of 3Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the PanguBot reference, the full census, and the JSON endpoint.