Blocklist · Zyte (open-source framework)

Sites that block Scrapy

706 of the 29,991 domains we have measured, 2%, disallow scrapy at the site root in robots.txt. Scrapy collects content that may be used to train a model.

Every domain here has withheld its content from Zyte (open-source framework)'s training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

706 of 29,991 measured domains (2%) disallow scrapy at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Scrapy", measured 2026-11-21. https://crawlcensus.com/blocked/scrapy

The list 706 domains

RankDomainTLDAI access score
8,763 mijnwoordenboek.nl .nl 73
8,830 portalsamorzadowy.pl .pl 69
8,854 nikkeibp.co.jp .jp 64
8,870 clarivate.com .com 70
8,897 rynekzdrowia.pl .pl 64
8,928 nottingham.ac.uk .uk 64
8,959 glamour.com .com 81
9,062 bluewin.ch .ch 66
9,069 wbur.org .org 81
9,149 forth.gr .gr 60
9,302 macworld.com .com 85
9,339 detroitnews.com .com 68
9,388 law.com .com 68
9,471 thewrap.com .com 87
9,718 diva-portal.org .org 49
9,885 wral.com .com 87
10,279 klingai.com .com 78
10,289 interestingengineering.com .com 87
10,291 usmagazine.com .com 87
10,345 inews.co.uk .uk 67
10,470 babycenter.com .com 62
10,552 self.com .com 68
10,575 getjobber.com .com 80
10,708 vogue.co.uk .uk 79
10,709 solarwinds.com .com 72
10,777 netshort.com .com 87
10,781 47news.jp .jp 67
10,826 bonappetit.com .com 82
10,852 embrapa.br .br 63
10,885 livestrong.com .com 80
10,894 augsburger-allgemeine.de .de 81
10,913 wyborcza.pl .pl 66
11,109 hamburg.de .de 61
11,127 fronius.com .com 75
11,147 intechopen.com .com 67
11,200 aftenposten.no .no 76
11,201 latercera.com .com 80
11,212 jdsupra.com .com 75
11,216 nola.com .com 64
11,217 businessinsider.de .de 76
11,242 teenvogue.com .com 82
11,244 stocktwits.com .com 82
11,349 posteo.de .de 67
11,367 abc7.com .com 81
11,390 pantone.com .com 78
11,405 svd.se .se 78
11,457 wnp.pl .pl 65
11,465 houstonchronicle.com .com 54
11,516 cults3d.com .com 91
11,610 allure.com .com 84
11,746 fextralife.com .com 87
11,827 timesunion.com .com 52
12,023 r7.com .com 86
12,055 nbcnewyork.com .com 33
12,310 elte.hu .hu 73
12,533 indystar.com .com 73
12,542 epicurious.com .com 64
12,607 joinhandshake.com .com 87
12,694 androidpolice.com .com 83
12,724 w3techs.com .com 84
12,743 extremetech.com .com 64
12,799 curbed.com .com 73
12,841 stltoday.com .com 80
12,972 noticiasaominuto.com .com 85
12,990 nybooks.com .com 80
13,005 simpleflying.com .com 83
13,169 csoonline.com .com 86
13,380 flourish.studio .studio 92
13,398 gq-magazine.co.uk .uk 81
13,451 skyscrapercity.com .com 43
13,579 barcelona.cat .cat 82
13,660 thrillist.com .com 70
13,667 dispatch.com .com 74
13,704 trademe.co.nz .nz 56
13,772 khan.co.kr .kr 72
13,825 heraldscotland.com .com 81
13,860 francebleu.fr .fr 87
13,871 ilex.by .by 59
13,879 tennessean.com .com 74
14,009 abc7chicago.com .com 81
14,055 observador.pt .pt 83
14,090 simon.com .com 31
14,168 ncaa.com .com 85
14,248 bhf.org.uk .uk 89
14,350 abc7ny.com .com 81
14,408 nbclosangeles.com .com 33
14,562 nbcchicago.com .com 62
14,709 dazeddigital.com .com 78
14,826 rockpapershotgun.com .com 58
14,945 asha.org .org 65
14,961 townnews.com .com 75
15,174 pcgamesn.com .com 83
15,218 statesman.com .com 51
15,230 mysanantonio.com .com 54
15,269 gamesindustry.biz .biz 57
15,293 abc7news.com .com 81
15,321 coursehero.com .com 76
15,352 aaas.org .org 65
15,372 letudiant.fr .fr 81
15,401 desmoinesregister.com .com 74
15,476 computerbild.de .de 81
15,557 cincinnati.com .com 74
15,583 idp.com .com 85
16,118 healio.com .com 69
16,223 smashwords.com .com 58
16,254 unil.ch .ch 83
16,377 decider.com .com 79
16,428 e24.no .no 78
16,444 hometalk.com .com 63
16,556 tourvisor.ru .ru 91
16,564 mapgenie.io .io 59
16,664 lightningmaps.org .org 51
16,738 cnbctv18.com .com 84
16,769 highsnobiety.com .com 75
16,796 postandcourier.com .com 77
16,803 buffalonews.com .com 80
16,814 whattoexpect.com .com 65
16,897 nbcbayarea.com .com 62
16,972 nbcwashington.com .com 60
17,042 northjersey.com .com 74
17,183 semafor.com .com 72
17,255 thegamer.com .com 83
17,266 letemps.ch .ch 74
17,432 kyoto-np.co.jp .jp 82
17,469 palmbeachpost.com .com 73
17,489 dukeupress.edu .edu 73
17,492 ina.fr .fr 76
17,522 daangn.com .com 71
17,536 seattlepi.com .com 54
17,554 abc13.com .com 79
17,694 manager-magazin.de .de 63
17,750 6abc.com .com 79
17,886 agrarheute.com .com 75
18,280 hoerzu.de .de 45
18,336 24chasa.bg .bg 88
18,338 telemundo.com .com 83
18,442 empireonline.com .com 62
18,687 romatoday.it .it 67
18,812 nbcphiladelphia.com .com 62
18,822 nbcdfw.com .com 33
18,870 scotusblog.com .com 83
18,883 courier-journal.com .com 74
18,978 dexerto.com .com 78
19,298 nbcmiami.com .com 60
19,314 milanotoday.it .it 67
19,541 tecconcursos.com.br .br 88
19,624 fukuishimbun.co.jp .jp 76
19,696 givemesport.com .com 83
19,860 zawya.com .com 80
19,882 geo.de .de 82
20,167 sorbonne-universite.fr .fr 66
20,337 onelook.com .com 51
20,362 thedodo.com .com 70
20,364 derwesten.de .de 83
20,419 01net.com .com 93
20,460 mt.co.kr .kr 93
20,467 23andme.com .com 98
21,038 mainpost.de .de 80
21,139 ledevoir.com .com 70
21,231 epochtimes.com .com 85
21,267 seher.no .no 70
21,320 gazetatema.net .net 50
21,564 plotek.pl .pl 75
21,586 madison.com .com 80
21,600 pathosting.com .com 74
21,707 khronos.org .org 68
21,736 slate.fr .fr 68
21,823 desy.de .de 62
21,864 xvideosincesto.com .com 80
21,968 playtube.tv .tv 70
21,999 dailyhive.com .com 74
22,007 vg247.com .com 58
22,016 canva.in .in 86
22,035 dcu.ie .ie 75
22,216 airliners.net .net 44
22,459 leofame.com .com 67
22,797 mondosanita.it .it 75
22,840 nbcsandiego.com .com 64
23,084 app.com .com 69
23,258 vi-hentai.pro .pro 57
23,613 nordbayern.de .de 81
23,964 pcwelt.de .de 78
24,014 askmen.com .com 47
24,102 knoxnews.com .com 74
24,141 cntraveller.com .com 81
24,163 kling.ai .ai 80
24,190 jacksonville.com .com 74
24,233 glamourmagazine.co.uk .uk 81
24,520 nbcboston.com .com 33
24,525 zin.ru .ru 41
24,567 newspicks.com .com 44
24,593 vogue.in .in 64
24,710 tricolor.ru .ru 65
24,754 aol.co.uk .uk 66
24,946 vogue.fr .fr 81
24,961 miaminewtimes.com .com 73
24,977 wired.it .it 83
25,028 uni-greifswald.de .de 85
25,031 franceculture.fr .fr 81
25,048 tokyo-np.co.jp .jp 81
PreviousPage 2 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Scrapy reference, the full census, and the JSON endpoint.