Blocklist · Zyte (open-source framework)

Sites that block Scrapy

706 of the 29,991 domains we have measured, 2%, disallow scrapy at the site root in robots.txt. Scrapy collects content that may be used to train a model.

Every domain here has withheld its content from Zyte (open-source framework)'s training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

706 of 29,991 measured domains (2%) disallow scrapy at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Scrapy", measured 2026-11-21. https://crawlcensus.com/blocked/scrapy

The list 706 domains

RankDomainTLDAI access score
25,184 kompasiana.com .com 85
25,451 omaha.com .com 80
25,478 franceinter.fr .fr 80
25,564 theadvocate.com .com 71
25,587 sfexaminer.com .com 69
25,597 matchendirect.fr .fr 86
25,638 truenetwork.ru .ru 70
25,756 ctinsider.com .com 54
25,776 zinio.com .com 76
25,839 delawareonline.com .com 74
25,903 heraldtribune.com .com 73
25,911 arkansasonline.com .com 71
26,114 democratandchronicle.com .com 73
26,284 daytondailynews.com .com 67
26,342 straightdope.com .com 76
26,379 brisbanetimes.com.au .au 75
26,411 lanit.ru .ru 69
26,474 mbta.com .com 64
26,637 brigitte.de .de 79
26,728 thueringen.de .de 77
26,751 eluniverso.com .com 76
26,853 tallahassee.com .com 74
26,935 fd.nl .nl 73
26,963 premiumoutlets.com .com 31
27,126 domserver.de .de 81
27,256 amny.com .com 83
27,344 ga.de .de 80
27,361 westfalen-blatt.de .de 81
27,422 affinity.co .co 95
27,425 carbuzz.com .com 85
27,467 wfmz.com .com 81
27,696 staradvertiser.com .com 84
27,719 mazdausa.com .com 76
27,723 geek.com .com 38
27,737 texasattorneygeneral.gov .gov 65
27,891 blogto.com .com 67
27,946 trustedshops.de .de 73
28,038 topspeed.com .com 81
28,048 udc.es .es 69
28,170 suedkurier.de .de 81
28,616 amzn.com .com 40
28,626 openfoodfacts.org .org 59
28,678 thenational.scot .scot 81
28,705 wellcomecollection.org .org 45
28,775 vogue.de .de 65
29,011 fraserinstitute.org .org 51
29,045 tt.com .com 75
29,048 gq-magazin.de .de 66
29,115 gp.se .se 78
29,139 u-cursos.cl .cl 51
29,173 tudosaladeaula.com .com 91
29,273 expressnews.com .com 54
29,293 ceu.edu .edu 58
29,390 kompas.id .id 64
29,402 mmafighting.com .com 68
29,510 uvigo.es .es 56
29,553 hankookilbo.com .com 84
29,567 ctpost.com .com 54
29,634 westword.com .com 77
29,692 piliapp.com .com 65
29,821 providencejournal.com .com 73
30,004 sfstandard.com .com 72
30,237 clarionledger.com .com 74
30,356 edaily.co.kr .kr 74
30,387 bsc.es .es 37
30,443 dobrzemieszkaj.pl .pl 61
30,500 routard.com .com 78
30,507 abc11.com .com 81
30,609 heinonline.org .org 46
30,672 thegameroom.org .org 91
30,703 wickedlocal.com .com 75
30,974 techadvisor.com .com 85
30,981 uni-passau.de .de 73
30,986 commercialappeal.com .com 74
30,992 huffingtonpost.it .it 83
31,023 phoenixnewtimes.com .com 71
31,032 howlongtobeat.com .com 49
31,176 edp24.co.uk .uk 79
31,248 utsystem.edu .edu 76
31,252 forumactif.com .com 83
31,282 vndb.org .org 60
31,320 lumendatabase.org .org 51
31,352 sportsbusinessjournal.com .com 53
31,443 saarbruecker-zeitung.de .de 82
31,499 bolasport.com .com 76
31,549 lohud.com .com 74
31,782 mastodon.online .online 49
31,786 thespec.com .com 64
31,796 winespectator.com .com 82
31,850 avsforum.com .com 43
32,169 journaldugeek.com .com 91
32,263 shacknews.com .com 73
32,267 theargus.co.uk .uk 79
32,388 rgj.com .com 73
32,437 hoopshype.com .com 70
32,561 skapiec.pl .pl 77
32,579 dallasobserver.com .com 77
32,580 planetadelibros.com .com 83
32,671 classicfm.com .com 83
32,718 athletic.net .net 68
32,790 tatler.com .com 82
32,803 oxfordmail.co.uk .uk 79
33,260 transfernow.net .net 92
33,411 lematin.ch .ch 84
33,494 theanalyst.com .com 88
33,562 myporntube.com .com 86
33,931 lancasteronline.com .com 79
33,970 spitogatos.gr .gr 39
34,097 vogue.es .es 65
34,387 breakingnews.ie .ie 71
34,445 monoskop.org .org 43
34,491 sfweekly.com .com 74
34,750 mailplus.co.uk .uk 44
34,818 securitm.ru .ru 92
34,822 milanofinanza.it .it 82
34,961 volksfreund.de .de 83
34,973 naplesnews.com .com 73
34,983 wbez.org .org 78
35,324 northdata.de .de 81
35,364 road.cc .cc 84
35,458 beritasatu.com .com 63
35,543 news-journalonline.com .com 73
35,625 enterprisedb.com .com 92
35,759 houseandgarden.co.uk .uk 81
35,800 wn.de .de 80
35,933 hrs.com .com 82
35,953 redflagdeals.com .com 44
36,325 kupikod.com .com 91
36,334 ytmnd.com .com 41
36,356 elgaronline.com .com 45
36,539 nd-aktuell.de .de 78
36,575 dailyecho.co.uk .uk 81
36,718 arynews.tv .tv 91
36,827 verdi.de .de 58
37,209 23andme.org .org 98
37,262 foodnavigator.com .com 88
37,303 lessentiel.lu .lu 84
37,305 hotcars.com .com 82
37,547 bt.no .no 77
38,007 wired.jp .jp 78
38,015 thetravel.com .com 81
38,118 abc30.com .com 81
38,129 11freunde.de .de 58
38,148 news-press.com .com 73
38,327 inforum.com .com 70
38,409 igt.com .com 48
38,507 bournemouthecho.co.uk .uk 79
38,519 kochbar.de .de 62
38,712 simplebooking.it .it 74
38,960 x-star.com .com 82
39,206 futurezone.de .de 86
39,210 thecentersquare.com .com 78
39,303 watoday.com.au .au 75
39,350 thehighline.org .org 77
39,369 santafenewmexican.com .com 76
39,393 zakzak.co.jp .jp 72
39,437 at-s.com .com 81
39,471 vwvortex.com .com 43
39,806 mercateo.com .com 48
39,824 werstreamt.es .es 59
39,950 dholding.ru .ru 56
40,095 walletinvestor.com .com 87
40,184 estantevirtual.com.br .br 91
40,305 seia.org .org 70
40,317 iza.ne.jp .jp 79
40,328 bntu.by .by 80
40,339 kompas.tv .tv 76
40,360 noen.at .at 79
40,413 dualshockers.com .com 83
40,495 yahoo.com.tw .tw 59
40,512 thetelegraphandargus.co.uk .uk 78
40,638 main-echo.de .de 64
40,678 aristeguinoticias.com .com 78
40,713 dwds.de .de 64
40,796 elpasotimes.com .com 73
40,874 grid.id .id 64
40,876 cntraveller.in .in 81
40,904 nesn.com .com 73
40,968 kontan.co.id .id 77
41,026 lnwshop.com .com 86
41,098 gqmagazine.fr .fr 82
41,121 lancashiretelegraph.co.uk .uk 78
41,204 elpuntavui.cat .cat 36
41,295 newson6.com .com 82
41,309 terminix.com .com 92
41,319 uwsp.edu .edu 90
41,336 archinect.com .com 68
41,519 greenme.it .it 82
41,542 oefb.at .at 50
41,647 cjonline.com .com 73
41,713 roanoke.com .com 80
41,736 eadt.co.uk .uk 79
41,826 defcon.org .org 66
41,872 vanityfair.fr .fr 66
41,947 vogue.co.jp .jp 65
41,958 theledger.com .com 74
41,962 okinawatimes.co.jp .jp 79
42,054 lecker.de .de 88
42,076 iex.nl .nl 48
42,112 news9.com .com 85
PreviousPage 3 of 4Next

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Scrapy reference, the full census, and the JSON endpoint.