# Domains that disallow Scrapy in robots.txt
# agent: scrapy
# generated: 2026-08-25T21:36:25.000Z
# count: 794
#
# What this count is, so it reconciles with the rate published elsewhere:
#   794 of 46919 domains with a known policy for this agent disallow it (1.7%).
#   Of those that also served a page, 683 of 30347 disallow it (2.3%).
#   https://crawlcensus.com/operator pages quote the second figure, because a site that refuses every
#   programmatic client is not evidence about AI policy. This file lists the first,
#   because a deny list must include a site whose robots.txt disallows you even when
#   its homepage did not answer our probe. Dividing this file's length by the corpus
#   size gives neither figure.
# measured by direct request, not inferred. Method: https://crawlcensus.com/methodology
# cursor: 1787467148.1480
# resume exactly here, no gap and no replay:
#   https://crawlcensus.com/agents/scrapy/changes.json?cursor=1787467148.1480
# this list is robots.txt only. Some domains additionally refuse you at the edge:
#   https://crawlcensus.com/api/v1/preflight
# licence: CC BY 4.0, attribute as "Source: Crawl Census (https://crawlcensus.com)"
01net.com
11freunde.de
15min.lt
20min.ch
23andme.com
23andme.org
24chasa.bg
47news.jp
4players.de
588ku.com
8newsnow.com
a1.net
aaas.org
abc27.com
abc4.com
advrider.com
adweek.com
affinity.co
afr.com
aftenbladet.no
aftenposten.no
aftonbladet.se
airliners.net
aktuality.sk
aladin.co.kr
alisweb.org
allgaeuer-zeitung.de
alltheweb.com
alrosa.ru
altavista.com
amazon.ae
amazon.ca
amazon.co.jp
amazon.co.uk
amazon.co.za
amazon.com
amazon.com.au
amazon.com.br
amazon.com.mx
amazon.com.tr
amazon.de
amazon.eg
amazon.es
amazon.fr
amazon.ie
amazon.in
amazon.it
amazon.jp
amazon.nl
amazon.pl
amazon.sa
amazon.se
amazon.sg
amazonvideo.com
amny.com
amzn.com
androidpolice.com
aol.co.uk
aol.com
aol.de
app.com
appian.com
archinect.com
argusleader.com
aristeguinoticias.com
arkansasonline.com
asahi.com
asha.org
askmen.com
athletics.app
augsburger-allgemeine.de
auto-tests.com
autobild.de
autoguide.com
autozeitung.de
avsforum.com
axios.com
azcentral.com
babycenter.com
babycentre.co.uk
bakersfield.com
barcablaugranes.com
barcelona.cat
bbc.co.uk
bbc.com
beaconjournal.com
benzinga.com
bestlawyers.com
bhf.org.uk
bild.de
bisnis.com
bizapedia.com
blackmagicdesign.com
blipfoto.com
blogto.com
bluewin.ch
bntu.by
bolasport.com
bonn.de
bournemouthecho.co.uk
breakingnews.ie
brigitte.de
brisbanetimes.com.au
broadinstitute.org
bsc.es
bt.no
burlingtonfreepress.com
businessinsider.com
businessinsider.de
bz-berlin.de
cagesideseats.com
candlepowerforums.com
canva.cn
canva.com
canva.in
capital.de
capitalfm.com
carbuzz.com
carter-cash.com
casadellibro.com
cbs17.com
centurylink.net
channel3000.com
chinatimes.com
chron.com
chroniclejournal.com
chrono24.co.uk
chrono24.com
chrono24.in
chugoku-np.co.jp
chunichi.co.jp
cincinnati.com
cio.com
citizen-times.com
cjonline.com
clarionledger.com
clarivate.com
classicfm.com
clutchpoints.com
cnbc.com
cnbctv18.com
cnet.com
cnn.com
cnnamador.com
codeberg.org
codeforces.com
collider.com
commercialappeal.com
computerbild.de
computerhope.com
computerwoche.de
computerworld.com
congress.gov
corporationwiki.com
corriere.it
courier-journal.com
coursehero.com
csmonitor.com
csoonline.com
ctee.com.tw
ctinsider.com
ctpost.com
curbed.com
dailycal.org
dailyecho.co.uk
dailygazette.com
dailyhive.com
dailymail.co.uk
dailymail.com
dallasnews.com
daytondailynews.com
dazeddigital.com
dcu.ie
dealereprocess.com
decider.com
defcon.org
delawareonline.com
democratandchronicle.com
dentsplysirona.com
derwesten.de
desertsun.com
desmoinesregister.com
detroitnews.com
dexerto.com
dholding.ru
dispatch.com
dobrzemieszkaj.pl
dualshockers.com
dukeupress.edu
dw.com
dwds.de
e24.no
eadt.co.uk
eater.com
ebsco.com
echo-news.co.uk
edaily.co.kr
edp24.co.uk
elgaronline.com
elpasotimes.com
elpuntavui.cat
elte.hu
eltern.de
eluniverso.com
empireonline.com
enterprisedb.com
eonline.com
epochtimes.com
essen-und-trinken.de
estantevirtual.com.br
eumetsat.int
eurekalert.org
eurogamer.de
eurogamer.net
everydayhealth.com
everyeye.it
expressnews.com
expresso.pt
extremetech.com
f95zone.to
falter.at
famousfix.com
fd.nl
fextralife.com
fintel.io
firstcry.com
fixya.com
flipboard.com
floridatoday.com
flourish.studio
flypobeda.ru
folhavitoria.com.br
foodnavigator.com
forth.gr
forumactif.com
fox2now.com
fox40.com
fox4kc.com
fox59.com
fox5sandiego.com
fox8.com
francebleu.fr
franceculture.fr
franceinter.fr
fraserinstitute.org
freeones.com
freep.com
fronius.com
fukuishimbun.co.jp
furaffinity.net
futurezone.de
ga.de
gainesville.com
gala.de
gala.fr
gamerant.com
gamesindustry.biz
gazeta.pl
gazetatema.net
gds.it
geek.com
gentoo.org
geo.de
getjobber.com
getpostman.com
givemesport.com
glasgowtimes.co.uk
gofeminin.de
govtrack.us
gp.se
greenme.it
greenvilleonline.com
grid.id
grindosaur.com
grubstreet.com
gunbark.dev
hamburg.de
hankookilbo.com
hdmall.co.th
healio.com
heinonline.org
helloasso.com
heraldscotland.com
heraldtribune.com
highsnobiety.com
hinative.com
hochi.news
hoerzu.de
homebrewtalk.com
hometalk.com
hoopshype.com
horseracingnation.com
hotcars.com
houstonchronicle.com
howlongtobeat.com
howtogeek.com
hrs.com
huffingtonpost.it
iacr.org
ibtimes.com
ici.fr
idp.com
iex.nl
ifixit.com
ign.com
igt.com
ilex.by
iltalehti.fi
imageshack.com
imageshack.us
ina.fr
indeed.com
indiamart.com
indiedb.com
indystar.com
inews.co.uk
inforum.com
infoworld.com
insider.com
intechopen.com
interestingengineering.com
investingchannel.com
iza.ne.jp
jacksonville.com
jamanetwork.com
japantravel.com
jdsupra.com
jiji.com
jogatina.com
joinhandshake.com
joongang.co.kr
journaldugeek.com
jsonline.com
jtbc.co.kr
jwz.org
kauppalehti.fi
kdvr.com
keloland.com
kfor.com
khon2.com
khronos.org
kittl.com
klingai.com
knoxnews.com
koin.com
kompas.com
kompas.id
kompas.tv
kompasiana.com
kontan.co.id
kotaku.com
kron4.com
krqe.com
ktar.com
ktla.com
kupikod.com
kxan.com
lagazzettadelmezzogiorno.it
lancashiretelegraph.co.uk
lancasteronline.com
lanit.ru
lansingstatejournal.com
lasicilia.it
lastampa.it
launchpad.net
law.com
lecker.de
leckerschmecker.me
ledevoir.com
lefigaro.fr
lematin.ch
leofame.com
lessentiel.lu
letemps.ch
letudiant.fr
levels.fyi
lifehacker.com
livemaster.ru
livestrong.com
lnwshop.com
lohud.com
lvm.de
macomnet.ru
macworld.com
mailonsunday.co.uk
mailplus.co.uk
main-echo.de
mainichi.jp
mainpost.de
makeuseof.com
mamamia.com.au
manager-magazin.de
mapgenie.io
mashable.com
matchendirect.fr
mathnet.ru
maxroll.gg
mazdausa.com
mbta.com
medicalxpress.com
medpagetoday.com
mercateo.com
meta.com
metro.co.uk
mijnwoordenboek.nl
milanofinanza.it
missouribotanicalgarden.org
mmafighting.com
mod.io
moddb.com
mondosanita.it
monoskop.org
montgomeryadvertiser.com
mora.jp
mos.ru
motorcycle.com
movieweb.com
mparticle.com
ms.now
msn.com
msnbc.com
mt.co.kr
mtv.fi
museodelprado.es
myanimelist.net
myfox8.com
mynorthwest.com
myporntube.com
mysanantonio.com
n-able.com
naiz.eus
naplesnews.com
nbc4i.com
nbcbayarea.com
nbcboston.com
nbcchicago.com
nbcdfw.com
nbclosangeles.com
nbcmiami.com
nbcnews.com
nbcnewyork.com
nbcphiladelphia.com
nbcsandiego.com
nbcwashington.com
ncaa.com
nd-aktuell.de
nefisyemektarifleri.com
nesn.com
netshort.com
nettavisen.no
news-journalonline.com
news-leader.com
news-postseven.com
news-press.com
news10.com
news24.com
news9.com
newsandstar.co.uk
newsbreak.com
newscientist.com
newsnationnow.com
newson6.com
newspapers.com
newspicks.com
newsweek.com
nhregister.com
nicematin.com
niche.com
nikkeibp.co.jp
nine.com.au
nishinippon.co.jp
nme.com
nn.de
noen.at
nola.com
nordbayern.de
northdata.com
northdata.de
northjersey.com
noticiasaominuto.com
nottingham.ac.uk
nrc.nl
nrdc.org
nrk.no
nrtool.st
numerosconsuerte.com.ar
nwaonline.com
nybooks.com
nymag.com
nypost.com
nyt.com
nyt.net
nytimes.com
nzherald.co.nz
nzz.ch
observador.pt
oefb.at
oilprice.com
okinawatimes.co.jp
oklahoman.com
olhardigital.com.br
omim.org
onelook.com
openfoodfacts.org
oru.se
outdooractive.com
oxfordmail.co.uk
pagesix.com
palmbeachpost.com
pantone.com
parismatch.com
pcmag.com
pcworld.com
phys.org
piliapp.com
pix11.com
pjstar.com
planetadelibros.com
playtube.tv
plotek.pl
pnj.com
pocket-lint.com
politico.eu
popsugar.com
portableapps.com
portalsamorzadowy.pl
posfie.com
postandcourier.com
postcrossing.com
posteo.de
postman.com
premiumoutlets.com
presse-citron.net
pricesmart.com
primevideo.com
providencejournal.com
pudelek.pl
quora.com
r7.com
radiofrance.fr
rainews.it
raiplay.it
raiplaysound.it
rcgroups.com
redflagdeals.com
refinery29.com
registerguard.com
renewableenergyworld.com
repubblica.it
restaurantbusinessonline.com
restofworld.org
reviewed.app
rexel.fr
rgj.com
rivm.nl
road.cc
roanoke.com
rockpapershotgun.com
rosebikes.de
routard.com
rp-online.de
rpp.pe
rtvslo.si
rynekzdrowia.pl
saarbruecker-zeitung.de
sankei.com
sanspo.com
santafenewmexican.com
savannahnow.com
sbnation.com
scotusblog.com
seacoastonline.com
seattlepi.com
securitm.ru
seekingalpha.com
seia.org
sekisuihouse.co.jp
sfchronicle.com
sfexaminer.com
sfgate.com
sfstandard.com
sfweekly.com
shacknews.com
sibur.ru
sicnoticias.pt
simon.com
simplebooking.it
simpleflying.com
sj-r.com
skapiec.pl
skyscrapercity.com
slate.com
slate.fr
smashwords.com
sme.sk
smh.com.au
snapchat.com
solarwinds.com
sorbonne-universite.fr
southwalesargus.co.uk
spiegel.de
spitogatos.gr
sport.de
sport.pl
sportingnews.com
sportler.com
sportsbusinessjournal.com
staradvertiser.com
statesman.com
statesmanjournal.com
stern.de
stocktwits.com
straightdope.com
suedkurier.de
suntimes.com
supercheats.com
svd.se
tallahassee.com
tcpalm.com
tcpdf.org
tds.net
tecconcursos.com.br
techadvisor.com
techbook.de
technologyreview.com
technorati.com
techxplore.com
telegram.com
telegraph.co.uk
telemundo.com
telstra.com
telstra.com.au
tempo.co
tennessean.com
terminix.com
texasattorneygeneral.gov
texasmonthly.com
tfm.co.jp
theadvocate.com
theage.com.au
theanalyst.com
theargus.co.uk
theathletic.com
theatlantic.com
thebulletin.org
thecentersquare.com
theconversation.com
theculturetrip.com
thecut.com
thedodo.com
thedrum.com
theepochtimes.com
thegamer.com
thegameroom.org
thegazette.com
theglobeandmail.com
thehill.com
theledger.com
themoviedb.org
thenational.scot
thenorthernecho.co.uk
theonion.com
therichest.com
thespec.com
thestar.com
thetelegraphandargus.co.uk
thetravel.com
theverge.com
theworlds50best.com
thewrap.com
thisismoney.co.uk
thrillist.com
thueringen.de
timesfreepress.com
timesunion.com
tmdb.org
today.com
today.it
todocoleccion.net
togetter.com
tokyo-np.co.jp
topekalibraries.info
topspeed.com
tourvisor.ru
townnews.com
trademe.co.nz
transfernow.net
travelbook.de
tricolor.ru
tricolor.tv
trinidadexpress.com
trustedshops.com
trustedshops.de
tt.com
tu.berlin
tudosaladeaula.com
tv2.dk
tv4play.se
tvmovie.de
u-cursos.cl
ugmk-telecom.ru
uni-greifswald.de
uni-hohenheim.de
uni-passau.de
unicaen.fr
unil.ch
unionleader.com
usatoday.com
usmagazine.com
utsystem.edu
uwsp.edu
vcstar.com
veikkaus.fi
verdi.de
vg.no
vg247.com
vi-hentai.pro
videocardz.com
vietnam.vn
vin.com
vndb.org
vnexpress.net
volksfreund.de
vox.com
vulture.com
vwvortex.com
w3techs.com
walletinvestor.com
wane.com
watoday.com.au
wavy.com
wbez.org
wbur.org
wdrb.com
weather.com
webofcomics.top
wellcomecollection.org
welt.de
werstreamt.es
westfalen-blatt.de
wfla.com
wfmz.com
wgntv.com
whatsapp.com
whatsapp.net
whattoexpect.com
who.int
wickedlocal.com
wikihow.com
wikihow.life
windstream.net
winelab.ru
winespectator.com
winknews.com
wivb.com
wkbn.com
wkrn.com
wn.de
wnp.pl
womansworld.com
woodtv.com
worcesternews.co.uk
workplace.com
worldsurfleague.com
wowhead.com
wpri.com
wral.com
wreg.com
wric.com
wtnh.com
wunderweib.de
wwlp.com
wyborcza.pl
wz.de
x-star.com
xda-developers.com
xe.gr
xvideosincesto.com
yahoo.com
yahoo.com.tw
yahoo.net
yle.fi
ylilauta.org
yna.co.kr
yorkpress.co.uk
yota.ru
yr.no
ytmnd.com
zakzak.co.jp
zawya.com
zdnet.com
zeczec.com
zetaglobal.net
zin.ru