Blocklist · Zyte (open-source framework)

Sites that block Scrapy

706 of the 29,991 domains we have measured, 2%, disallow scrapy at the site root in robots.txt. Scrapy collects content that may be used to train a model.

Every domain here has withheld its content from Zyte (open-source framework)'s training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

706 of 29,991 measured domains (2%) disallow scrapy at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Scrapy", measured 2026-11-21. https://crawlcensus.com/blocked/scrapy

The list 706 domains

RankDomainTLDAI access score
26 amazon.com .com 40
58 yahoo.com .com 67
62 msn.com .com 41
121 amazonvideo.com .com 55
190 cnn.com .com 50
207 bbc.com .com 76
226 bbc.co.uk .uk 77
246 who.int .int 90
260 launchpad.net .net 52
282 weather.com .com 86
318 amazon.co.uk .uk 40
377 amazon.de .de 40
426 businessinsider.com .com 93
457 cnbc.com .com 54
494 amazon.co.jp .jp 40
499 aol.com .com 66
533 wired.com .com 81
556 alrosa.ru .ru 68
609 amazon.fr .fr 40
623 amazon.co.za .za 40
627 dailymail.co.uk .uk 61
635 amazon.ca .ca 40
659 amazon.in .in 40
662 cnet.com .com 79
681 dailymail.com .com 61
682 primevideo.com .com 71
706 amazon.it .it 40
725 amazon.es .es 40
763 bild.de .de 81
812 amazon.com.au .au 40
815 theverge.com .com 77
823 nbcnews.com .com 59
833 amazon.com.br .br 40
834 lefigaro.fr .fr 71
942 repubblica.it .it 85
955 corriere.it .it 63
1,019 theconversation.com .com 62
1,032 amazon.com.mx .mx 40
1,042 spiegel.de .de 64
1,054 n-tv.de .de 88
1,083 nypost.com .com 81
1,189 aftonbladet.se .se 77
1,196 vnexpress.net .net 88
1,208 vg.no .no 79
1,223 furaffinity.net .net 39
1,272 vietnam.vn .vn 74
1,300 f95zone.to .to 59
1,305 dw.com .com 42
1,376 dagbladet.no .no 74
1,394 usnews.com .com 83
1,409 iltalehti.fi .fi 58
1,793 newyorker.com .com 73
1,831 zdnet.com .com 76
1,835 wikihow.com .com 76
1,984 themoviedb.org .org 74
1,993 ign.com .com 62
2,006 amazon.eg .eg 40
2,010 mashable.com .com 61
2,325 nrk.no .no 69
2,346 slate.com .com 80
2,423 pcmag.com .com 63
2,430 sfgate.com .com 54
2,508 yr.no .no 64
2,530 smh.com.au .au 82
2,561 technologyreview.com .com 82
2,691 newscientist.com .com 76
2,723 yumpu.com .com 30
2,838 mastodon.social .social 49
2,899 vanityfair.it .it 77
2,969 today.com .com 62
2,975 imageshack.us .us 71
2,989 vogue.com .com 82
3,116 newsbreak.com .com 91
3,198 metro.co.uk .uk 59
3,208 n-able.com .com 84
3,276 myanimelist.net .net 70
3,286 flipboard.com .com 64
3,342 lifehacker.com .com 65
3,368 nzherald.co.nz .nz 56
3,371 nymag.com .com 74
3,457 telstra.com .com 88
3,463 postman.com .com 92
3,499 amazon.se .se 40
3,520 amazon.ae .ae 40
3,524 chron.com .com 54
3,533 vanityfair.com .com 81
3,595 politico.eu .eu 62
3,598 amazon.nl .nl 40
3,603 macomnet.ru .ru 63
3,642 yahoo.net .net 67
3,771 yle.fi .fi 85
3,800 ebsco.com .com 65
3,869 pcworld.com .com 84
3,909 aol.de .de 65
3,916 news24.com .com 56
4,014 eurekalert.org .org 48
4,061 screenrant.com .com 84
4,065 seekingalpha.com .com 91
4,083 stern.de .de 83
4,108 howtogeek.com .com 84
4,161 kompas.com .com 77
4,354 computerworld.com .com 83
4,497 gazeta.pl .pl 74
4,512 amazon.com.tr .tr 40
4,551 yna.co.kr .kr 87
4,592 lastampa.it .it 81
4,684 msnbc.com .com 75
4,753 xda-developers.com .com 83
4,766 adweek.com .com 69
4,791 collider.com .com 83
4,961 20min.ch .ch 84
5,037 imageshack.com .com 71
5,068 gq.com .com 81
5,096 nine.com.au .au 71
5,146 everydayhealth.com .com 60
5,153 theage.com.au .au 75
5,223 oilprice.com .com 90
5,251 dallasnews.com .com 54
5,272 depositphotos.com .com 63
5,274 csmonitor.com .com 76
5,297 chinatimes.com .com 64
5,310 kotaku.com .com 89
5,323 cbr.com .com 83
5,395 makeuseof.com .com 84
5,455 gamerant.com .com 81
5,463 rainews.it .it 53
5,523 architecturaldigest.com .com 83
5,547 amazon.ie .ie 40
5,678 dn.se .se 60
5,684 theepochtimes.com .com 80
5,726 chefkoch.de .de 81
5,752 sankei.com .com 76
5,788 ibtimes.com .com 84
5,879 cio.com .com 86
5,895 movieweb.com .com 84
5,904 windstream.net .net 53
5,912 pagesix.com .com 80
5,948 suntimes.com .com 81
5,993 eonline.com .com 81
6,006 insider.com .com 94
6,032 rtl.de .de 85
6,089 azcentral.com .com 74
6,109 fsf.org .org 84
6,185 trustedshops.com .com 87
6,194 ifixit.com .com 73
6,228 sbnation.com .com 69
6,245 raiplay.it .it 55
6,246 eurogamer.net .net 57
6,288 vulture.com .com 72
6,352 tv2.dk .dk 74
6,356 freep.com .com 73
6,361 ici.fr .fr 87
6,473 sfchronicle.com .com 54
6,504 telstra.com.au .au 86
6,512 telegram.com .com 73
6,522 eater.com .com 59
6,544 rtvslo.si .si 65
6,557 sportingnews.com .com 82
6,593 wired.co.uk .uk 64
6,702 infoworld.com .com 83
6,737 altavista.com .com 53
6,808 chunichi.co.jp .jp 71
6,816 maxroll.gg .gg 59
6,833 bergfex.at .at 81
6,895 togetter.com .com 73
6,948 nicematin.com .com 77
7,042 cntraveler.com .com 82
7,082 aktuality.sk .sk 85
7,106 nrc.nl .nl 84
7,109 military.com .com 84
7,118 nme.com .com 81
7,119 rp-online.de .de 79
7,127 hani.co.kr .kr 67
7,178 sanspo.com .com 77
7,239 gala.de .de 76
7,247 nishinippon.co.jp .jp 69
7,268 nzz.ch .ch 76
7,345 rpp.pe .pe 90
7,372 15min.lt .lt 74
7,423 jiji.com .com 81
7,455 amazon.sg .sg 40
7,465 nationalgeographic.org .org 82
7,531 theonion.com .com 84
7,540 freeones.com .com 71
7,562 nettavisen.no .no 63
7,738 pitchfork.com .com 81
7,821 gala.fr .fr 90
7,856 popsugar.com .com 75
7,863 refinery29.com .com 78
7,941 sport.pl .pl 75
7,947 cnnamador.com .com 74
7,981 afr.com .com 72
8,094 nefisyemektarifleri.com .com 76
8,263 hochi.news .news 78
8,380 veikkaus.fi .fi 67
8,435 thecut.com .com 84
8,484 sciencenews.org .org 86
8,502 a1.net .net 77
8,533 blackmagicdesign.com .com 72
8,755 pudelek.pl .pl 89
Page 1 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Scrapy reference, the full census, and the JSON endpoint.