Blocklist · Zyte (open-source framework)

Sites that block Scrapy

706 of the 29,991 domains we have measured, 2%, disallow scrapy at the site root in robots.txt. Scrapy collects content that may be used to train a model.

Every domain here has withheld its content from Zyte (open-source framework)'s training crawl. That is a separate decision from appearing in the operator's answers, which is governed by a different token.

The measurement

706 of 29,991 measured domains (2%) disallow scrapy at the site root in robots.txt. Every domain named on this page was fetched directly and its robots.txt parsed against RFC 9309; none of it is inferred from a third-party index.

Measured 2026-11-21 by direct request to every domain in the corpus. Method: how this is measured. Reuse under CC BY 4.0 with attribution.

Cite as: Crawl Census, "Sites that block Scrapy", measured 2026-11-21. https://crawlcensus.com/blocked/scrapy

The list 706 domains

RankDomainTLDAI access score
42,113 glasgowtimes.co.uk .uk 79
42,160 tcpalm.com .com 73
42,250 dentsplysirona.com .com 73
42,260 rcgroups.com .com 41
42,262 advrider.com .com 41
42,614 citizen-times.com .com 74
42,651 computerwoche.de .de 85
42,666 nhregister.com .com 54
42,698 sol.no .no 74
42,835 iodonna.it .it 86
42,862 horseracingnation.com .com 72
42,935 ugmk-telecom.ru .ru 63
43,132 jtbc.co.kr .kr 70
43,207 tv4play.se .se 77
43,268 candlepowerforums.com .com 61
43,302 seacoastonline.com .com 69
43,548 tvmovie.de .de 89
43,587 admagazine.fr .fr 81
43,616 appian.com .com 77
43,633 rexel.fr .fr 69
43,909 gipuzkoa.eus .eus 70
43,973 mathnet.ru .ru 66
43,986 nn.de .de 83
44,026 gainesville.com .com 66
44,116 eevblog.com .com 76
44,183 barcablaugranes.com .com 68
44,255 cagesideseats.com .com 72
44,366 capitalfm.com .com 80
44,646 zeczec.com .com 76
44,723 dailygazette.com .com 80
44,807 gofeminin.de .de 74
44,837 winelab.ru .ru 72
44,874 news-leader.com .com 62
44,908 mora.jp .jp 45
44,928 wgt.com .com 80
44,930 jogatina.com .com 80
44,968 savannahnow.com .com 64
45,002 babycentre.co.uk .uk 62
45,133 northdata.com .com 81
45,158 autoguide.com .com 87
45,159 lansingstatejournal.com .com 62
45,240 vogue.mx .mx 65
45,284 athletics.app .app 65
45,316 eumetsat.int .int 80
45,390 tupalo.com .com 57
45,452 landwirt.com .com 94
45,571 wz.de .de 76
45,578 chugoku-np.co.jp .jp 80
45,604 beaconjournal.com .com 66
45,622 famousfix.com .com 59
45,680 greenvilleonline.com .com 62
45,762 autozeitung.de .de 86
45,846 amazon.jp .jp 40
45,918 revistagq.com .com 81
45,989 aftenbladet.no .no 78
46,045 mod.io .io 51
46,168 bizapedia.com .com 60
46,209 rosebikes.de .de 81
46,231 registerguard.com .com 66
46,342 pjstar.com .com 66
46,360 pnj.com .com 62
46,560 raiplaysound.it .it 59
46,635 wikihow.life .life 76
46,753 grfc.ru .ru 50
46,923 alisweb.org .org 39
46,990 fixya.com .com 43
47,173 eurogamer.de .de 57
47,184 lvm.de .de 94
47,346 lasicilia.it .it 86
47,455 folhavitoria.com.br .br 78
47,492 kittl.com .com 91
47,512 bestlawyers.com .com 89
47,526 allgaeuer-zeitung.de .de 81
47,624 lacucinaitaliana.it .it 81
47,796 topekalibraries.info .info 46
47,884 vcstar.com .com 62
48,046 trinidadexpress.com .com 74
48,074 grubstreet.com .com 81
48,251 sekisuihouse.co.jp .jp 66
48,444 vogue.com.tw .tw 81
48,550 argusleader.com .com 62
48,689 naiz.eus .eus 75
48,777 mtv.fi .fi 76
48,868 tfm.co.jp .jp 76
49,057 institutionalinvestor.com .com 77
49,061 posfie.com .com 73
49,067 gqitalia.it .it 64
49,192 therichest.com .com 84
49,194 mailonsunday.co.uk .uk 61
49,271 eltern.de .de 72
49,420 gds.it .it 77
49,501 sj-r.com .com 64
49,574 montgomeryadvertiser.com .com 62
49,695 pricesmart.com .com 61
49,802 pazar3.mk .mk 73
49,879 fandomwire.com .com 67
50,278 burlingtonfreepress.com .com 62
50,367 blipfoto.com .com 57
50,520 womansworld.com .com 85
50,598 dailycal.org .org 74
50,684 gq.com.mx .mx 72
50,916 motorcycle.com .com 83
50,919 japantravel.com .com 79
50,933 nwaonline.com .com 73
50,982 nrtool.st .st 54
- tds.net .net 53
PreviousPage 4 of 4

Rank is the domain's position in the popularity list used to seed the corpus; a dash means it was scanned on request rather than seeded. Score is the site's overall AI access score, which blocking depresses but does not by itself determine.

Check a specific domain

See also the Scrapy reference, the full census, and the JSON endpoint.