Change log

Who changed their mind

Crawler policy is edited quietly, usually with no announcement. Every scan is compared against the previous one for that domain, and any difference is recorded here permanently.

300

recorded changes

114

crawlers newly blocked

crawlers newly allowed

Every recorded change newest first

block lindependant.fr started blocking GPTBot 1 minute ago
block lindependant.fr started blocking ClaudeBot 1 minute ago
block lindependant.fr started blocking anthropic-ai 1 minute ago
block lindependant.fr started blocking Google-Extended 1 minute ago
block lindependant.fr started blocking PerplexityBot 1 minute ago
block lindependant.fr started blocking FacebookBot 1 minute ago
block lindependant.fr started blocking Bytespider 1 minute ago
block lindependant.fr started blocking Amazonbot 1 minute ago
block lindependant.fr started blocking CCBot 1 minute ago
block lindependant.fr started blocking Diffbot 1 minute ago
block lindependant.fr started blocking omgili 1 minute ago
block lindependant.fr started blocking omgilibot 1 minute ago
block lindependant.fr started blocking cohere-ai 1 minute ago
block lindependant.fr started blocking YouBot 1 minute ago
block lindependant.fr started blocking SemrushBot-OCOB 1 minute ago
score ridewithvia.com changed score (76 → 86) 1 minute ago
score bata.com changed score (85 → 71) 1 minute ago
block statnews.com started blocking GPTBot 1 minute ago
block statnews.com started blocking ClaudeBot 1 minute ago
block statnews.com started blocking anthropic-ai 1 minute ago
block statnews.com started blocking Google-Extended 1 minute ago
block statnews.com started blocking GoogleOther 1 minute ago
block statnews.com started blocking Applebot-Extended 1 minute ago
block statnews.com started blocking Meta-ExternalAgent 1 minute ago
block statnews.com started blocking FacebookBot 1 minute ago
block statnews.com started blocking Bytespider 1 minute ago
block statnews.com started blocking Amazonbot 1 minute ago
block statnews.com started blocking CCBot 1 minute ago
block statnews.com started blocking Diffbot 1 minute ago
block statnews.com started blocking omgili 1 minute ago
block statnews.com started blocking omgilibot 1 minute ago
block statnews.com started blocking cohere-ai 1 minute ago
block statnews.com started blocking cohere-training-data-crawler 1 minute ago
block statnews.com started blocking PanguBot 1 minute ago
block statnews.com started blocking Timpibot 1 minute ago
score eldebate.com changed score (59 → 86) 4 minutes ago
edge-serve biography.com stopped refusing at the edge GPTBot 4 minutes ago
edge-refuse biography.com started refusing at the edge ClaudeBot 4 minutes ago
edge-serve notepad-plus-plus.org stopped refusing at the edge ClaudeBot 7 minutes ago
edge-serve notepad-plus-plus.org stopped refusing at the edge PerplexityBot 7 minutes ago
score eprocure.gov.bd changed score (64 → 46) 13 minutes ago
block jiji.com started blocking GPTBot 13 minutes ago
block jiji.com started blocking OAI-SearchBot 13 minutes ago
block jiji.com started blocking ChatGPT-User 13 minutes ago
block jiji.com started blocking ClaudeBot 13 minutes ago
block jiji.com started blocking Claude-User 13 minutes ago
block jiji.com started blocking Claude-SearchBot 13 minutes ago
block jiji.com started blocking anthropic-ai 13 minutes ago
block jiji.com started blocking Google-Extended 13 minutes ago
block jiji.com started blocking Applebot-Extended 13 minutes ago
block jiji.com started blocking PerplexityBot 13 minutes ago
block jiji.com started blocking Perplexity-User 13 minutes ago
block jiji.com started blocking Meta-ExternalAgent 13 minutes ago
block jiji.com started blocking Meta-ExternalFetcher 13 minutes ago
block jiji.com started blocking FacebookBot 13 minutes ago
block jiji.com started blocking Bytespider 13 minutes ago
block jiji.com started blocking Amazonbot 13 minutes ago
block jiji.com started blocking CCBot 13 minutes ago
block jiji.com started blocking Diffbot 13 minutes ago
block jiji.com started blocking omgili 13 minutes ago
block jiji.com started blocking omgilibot 13 minutes ago
block jiji.com started blocking cohere-ai 13 minutes ago
block jiji.com started blocking cohere-training-data-crawler 13 minutes ago
block jiji.com started blocking Timpibot 13 minutes ago
block jiji.com started blocking Scrapy 13 minutes ago
edge-refuse draftkings.com started refusing at the edge GPTBot 13 minutes ago
edge-refuse draftkings.com started refusing at the edge OAI-SearchBot 13 minutes ago
edge-refuse draftkings.com started refusing at the edge ClaudeBot 13 minutes ago
edge-refuse draftkings.com started refusing at the edge PerplexityBot 13 minutes ago
block outsideonline.com started blocking GPTBot 16 minutes ago
edge-refuse outsideonline.com started refusing at the edge GPTBot 16 minutes ago
block outsideonline.com started blocking ClaudeBot 16 minutes ago
edge-refuse outsideonline.com started refusing at the edge ClaudeBot 16 minutes ago
block outsideonline.com started blocking anthropic-ai 16 minutes ago
block outsideonline.com started blocking Applebot-Extended 16 minutes ago
block outsideonline.com started blocking Meta-ExternalAgent 16 minutes ago
block outsideonline.com started blocking FacebookBot 16 minutes ago
block outsideonline.com started blocking Bytespider 16 minutes ago
block outsideonline.com started blocking Amazonbot 16 minutes ago
block outsideonline.com started blocking CCBot 16 minutes ago
block outsideonline.com started blocking Diffbot 16 minutes ago
block outsideonline.com started blocking omgili 16 minutes ago
block outsideonline.com started blocking omgilibot 16 minutes ago
block outsideonline.com started blocking AI2Bot 16 minutes ago
block outsideonline.com started blocking cohere-ai 16 minutes ago
block outsideonline.com started blocking cohere-training-data-crawler 16 minutes ago
block outsideonline.com started blocking Timpibot 16 minutes ago
block outsideonline.com started blocking ImagesiftBot 16 minutes ago
score outsideonline.com changed score (54 → 79) 16 minutes ago
edge-serve delish.com stopped refusing at the edge GPTBot 16 minutes ago
score virgilio.it changed score (59 → 87) 19 minutes ago
score airasia.com changed score (73 → 62) 19 minutes ago
unblock bandlab.com unblocked GPTBot 22 minutes ago
unblock bandlab.com unblocked ClaudeBot 22 minutes ago
edge-refuse chinaz.com started refusing at the edge GPTBot 22 minutes ago
llms added eltiempo.es published an llms.txt 31 minutes ago
score eltiempo.es changed score (47 → 66) 31 minutes ago
unblock anota.ai unblocked GPTBot 31 minutes ago
unblock anota.ai unblocked PerplexityBot 31 minutes ago
llms added anota.ai published an llms.txt 31 minutes ago
score anota.ai changed score (88 → 76) 31 minutes ago
unblock fakt.pl unblocked Meta-ExternalAgent 33 minutes ago
unblock fakt.pl unblocked Meta-WebIndexer 33 minutes ago
unblock fakt.pl unblocked Bytespider 33 minutes ago
unblock fakt.pl unblocked Amazonbot 33 minutes ago
unblock fakt.pl unblocked Amzn-SearchBot 33 minutes ago
unblock fakt.pl unblocked Amzn-User 33 minutes ago
unblock fakt.pl unblocked Diffbot 33 minutes ago
unblock fakt.pl unblocked omgilibot 33 minutes ago
unblock fakt.pl unblocked cohere-ai 33 minutes ago
unblock fakt.pl unblocked YouBot 33 minutes ago
unblock fakt.pl unblocked Timpibot 33 minutes ago
score 24sata.hr changed score (52 → 85) 34 minutes ago
block gala.de started blocking Google-CloudVertexBot 37 minutes ago
score sling.com changed score (53 → 75) 40 minutes ago
block nypl.org started blocking GPTBot 46 minutes ago
block nypl.org started blocking Bytespider 46 minutes ago
score zus.pl changed score (46 → 74) 46 minutes ago
edge-refuse worldatlas.com started refusing at the edge GPTBot 49 minutes ago
edge-refuse worldatlas.com started refusing at the edge OAI-SearchBot 49 minutes ago
edge-refuse worldatlas.com started refusing at the edge ClaudeBot 49 minutes ago
edge-refuse worldatlas.com started refusing at the edge PerplexityBot 49 minutes ago
score worldatlas.com changed score (84 → 67) 49 minutes ago
llms added drogariasaopaulo.com.br published an llms.txt 52 minutes ago
score drogariasaopaulo.com.br changed score (59 → 80) 52 minutes ago
edge-refuse hani.co.kr started refusing at the edge GPTBot 52 minutes ago
edge-refuse hani.co.kr started refusing at the edge OAI-SearchBot 52 minutes ago
edge-refuse hani.co.kr started refusing at the edge ClaudeBot 52 minutes ago
edge-refuse hani.co.kr started refusing at the edge PerplexityBot 52 minutes ago
score hani.co.kr changed score (75 → 58) 52 minutes ago
score tcd.ie changed score (47 → 72) 52 minutes ago
edge-refuse mambu.com started refusing at the edge GPTBot 52 minutes ago
edge-refuse mambu.com started refusing at the edge OAI-SearchBot 52 minutes ago
edge-refuse mambu.com started refusing at the edge ClaudeBot 52 minutes ago
edge-refuse mambu.com started refusing at the edge PerplexityBot 52 minutes ago
score mambu.com changed score (76 → 59) 52 minutes ago
block nrc.nl started blocking GPTBot 58 minutes ago
block nrc.nl started blocking OAI-SearchBot 58 minutes ago
block nrc.nl started blocking ChatGPT-User 58 minutes ago
block nrc.nl started blocking ClaudeBot 58 minutes ago
block nrc.nl started blocking Claude-User 58 minutes ago
block nrc.nl started blocking Claude-SearchBot 58 minutes ago
block nrc.nl started blocking anthropic-ai 58 minutes ago
block nrc.nl started blocking Google-Extended 58 minutes ago
block nrc.nl started blocking Google-CloudVertexBot 58 minutes ago
block nrc.nl started blocking GoogleOther 58 minutes ago
block nrc.nl started blocking Applebot-Extended 58 minutes ago
block nrc.nl started blocking PerplexityBot 58 minutes ago
block nrc.nl started blocking Perplexity-User 58 minutes ago
block nrc.nl started blocking Meta-ExternalAgent 58 minutes ago
block nrc.nl started blocking Bytespider 58 minutes ago
block nrc.nl started blocking TikTokSpider 58 minutes ago
block nrc.nl started blocking Amazonbot 58 minutes ago
block nrc.nl started blocking CCBot 58 minutes ago
block nrc.nl started blocking Diffbot 58 minutes ago
block nrc.nl started blocking omgili 58 minutes ago
block nrc.nl started blocking omgilibot 58 minutes ago
block nrc.nl started blocking AI2Bot 58 minutes ago
block nrc.nl started blocking cohere-ai 58 minutes ago
block nrc.nl started blocking cohere-training-data-crawler 58 minutes ago
block nrc.nl started blocking MistralAI-User 58 minutes ago
block nrc.nl started blocking DuckAssistBot 58 minutes ago
block nrc.nl started blocking YouBot 58 minutes ago
block nrc.nl started blocking PanguBot 58 minutes ago
block nrc.nl started blocking Timpibot 58 minutes ago
block nrc.nl started blocking ImagesiftBot 58 minutes ago
block nrc.nl started blocking Kangaroo Bot 58 minutes ago
block nrc.nl started blocking SemrushBot-OCOB 58 minutes ago
block nrc.nl started blocking Scrapy 58 minutes ago
edge-serve hvg.hu stopped refusing at the edge GPTBot 1 hour ago
edge-serve hvg.hu stopped refusing at the edge ClaudeBot 1 hour ago
score hvg.hu changed score (74 → 48) 1 hour ago
edge-serve cntraveler.com stopped refusing at the edge GPTBot 1 hour ago
edge-serve cntraveler.com stopped refusing at the edge OAI-SearchBot 1 hour ago
edge-serve cntraveler.com stopped refusing at the edge PerplexityBot 1 hour ago
score cntraveler.com changed score (70 → 82) 1 hour ago
llms added jbhifi.com.au published an llms.txt 1 hour ago
score jbhifi.com.au changed score (53 → 71) 1 hour ago
llms removed dm.de removed its llms.txt 1 hour ago
score dm.de changed score (64 → 52) 1 hour ago
edge-refuse discovermagazine.com started refusing at the edge ClaudeBot 1 hour ago
edge-refuse discovermagazine.com started refusing at the edge PerplexityBot 1 hour ago
score kino-teatr.ru changed score (41 → 51) 1 hour ago
edge-refuse elperiodico.com started refusing at the edge OAI-SearchBot 1 hour ago
score elperiodico.com changed score (56 → 78) 1 hour ago
score covers.com changed score (62 → 92) 1 hour ago
score ajc.com changed score (50 → 86) 1 hour ago
block nicematin.com started blocking MistralAI-Index 1 hour ago
edge-serve countryliving.com stopped refusing at the edge GPTBot 1 hour ago
llms added mql5.com published an llms.txt 1 hour ago
score williamhill.com changed score (59 → 74) 1 hour ago
edge-refuse dome9.com started refusing at the edge GPTBot 1 hour ago
edge-refuse dome9.com started refusing at the edge OAI-SearchBot 1 hour ago
edge-refuse dome9.com started refusing at the edge ClaudeBot 1 hour ago
edge-refuse dome9.com started refusing at the edge PerplexityBot 1 hour ago
score dome9.com changed score (92 → 75) 1 hour ago
score alberta.ca changed score (87 → 49) 1 hour ago
edge-serve akc.org stopped refusing at the edge ClaudeBot 1 hour ago
block maxroll.gg started blocking Amzn-User 1 hour ago
block maxroll.gg started blocking MistralAI-Index 1 hour ago
unblock maxroll.gg unblocked Kangaroo Bot 1 hour ago
edge-serve ekitan.com stopped refusing at the edge ClaudeBot 1 hour ago
edge-serve ekitan.com stopped refusing at the edge PerplexityBot 1 hour ago
edge-refuse icm.edu.pl started refusing at the edge GPTBot 1 hour ago
edge-refuse icm.edu.pl started refusing at the edge OAI-SearchBot 1 hour ago
edge-refuse icm.edu.pl started refusing at the edge ClaudeBot 1 hour ago
edge-refuse icm.edu.pl started refusing at the edge PerplexityBot 1 hour ago
score icm.edu.pl changed score (62 → 45) 1 hour ago
score stylevana.com changed score (65 → 82) 1 hour ago
score dafiti.com.br changed score (46 → 86) 1 hour ago
edge-refuse freewebs.com started refusing at the edge GPTBot 1 hour ago
edge-refuse freewebs.com started refusing at the edge OAI-SearchBot 1 hour ago
edge-refuse freewebs.com started refusing at the edge ClaudeBot 1 hour ago
edge-refuse freewebs.com started refusing at the edge PerplexityBot 1 hour ago
score freewebs.com changed score (73 → 56) 1 hour ago
score units.it changed score (46 → 75) 2 hours ago
score consumerfinance.gov changed score (58 → 77) 2 hours ago
score europapress.es changed score (46 → 74) 2 hours ago
score fws.gov changed score (54 → 71) 2 hours ago
score vistarsagency.com changed score (78 → 91) 2 hours ago
edge-serve jornada.com.mx stopped refusing at the edge GPTBot 2 hours ago
edge-serve jornada.com.mx stopped refusing at the edge ClaudeBot 2 hours ago
score jornada.com.mx changed score (70 → 50) 2 hours ago
edge-serve narvar.com stopped refusing at the edge ClaudeBot 2 hours ago
edge-refuse interno.gov.it started refusing at the edge GPTBot 2 hours ago
edge-refuse interno.gov.it started refusing at the edge OAI-SearchBot 2 hours ago
edge-refuse interno.gov.it started refusing at the edge ClaudeBot 2 hours ago
edge-refuse interno.gov.it started refusing at the edge PerplexityBot 2 hours ago
edge-refuse yellowpages.ca started refusing at the edge GPTBot 2 hours ago
edge-refuse yellowpages.ca started refusing at the edge OAI-SearchBot 2 hours ago
edge-refuse yellowpages.ca started refusing at the edge ClaudeBot 2 hours ago
edge-refuse yellowpages.ca started refusing at the edge PerplexityBot 2 hours ago
edge-serve procyclingstats.com stopped refusing at the edge OAI-SearchBot 2 hours ago
charge-start procyclingstats.com started charging OAI-SearchBot 2 hours ago
edge-serve procyclingstats.com stopped refusing at the edge PerplexityBot 2 hours ago
charge-start procyclingstats.com started charging PerplexityBot 2 hours ago
score racingpost.com changed score (59 → 80) 2 hours ago
edge-serve wired.co.uk stopped refusing at the edge GPTBot 2 hours ago
edge-serve wired.co.uk stopped refusing at the edge OAI-SearchBot 2 hours ago
edge-serve wired.co.uk stopped refusing at the edge ClaudeBot 2 hours ago
edge-serve wired.co.uk stopped refusing at the edge PerplexityBot 2 hours ago
score wired.co.uk changed score (64 → 81) 2 hours ago
edge-serve jorudan.co.jp stopped refusing at the edge GPTBot 2 hours ago
edge-serve jorudan.co.jp stopped refusing at the edge OAI-SearchBot 2 hours ago
edge-serve jorudan.co.jp stopped refusing at the edge ClaudeBot 2 hours ago
edge-serve jorudan.co.jp stopped refusing at the edge PerplexityBot 2 hours ago
score jorudan.co.jp changed score (34 → 51) 2 hours ago
edge-serve osf.io stopped refusing at the edge OAI-SearchBot 2 hours ago
edge-serve osf.io stopped refusing at the edge ClaudeBot 2 hours ago
edge-serve osf.io stopped refusing at the edge PerplexityBot 2 hours ago
score osf.io changed score (31 → 44) 2 hours ago
edge-serve goldprice.org stopped refusing at the edge ClaudeBot 2 hours ago
unblock sofifa.com unblocked GPTBot 2 hours ago
edge-serve sofifa.com stopped refusing at the edge GPTBot 2 hours ago
edge-serve sofifa.com stopped refusing at the edge OAI-SearchBot 2 hours ago
unblock sofifa.com unblocked ClaudeBot 2 hours ago
edge-serve sofifa.com stopped refusing at the edge ClaudeBot 2 hours ago
unblock sofifa.com unblocked Google-Extended 2 hours ago
unblock sofifa.com unblocked Applebot-Extended 2 hours ago
edge-serve sofifa.com stopped refusing at the edge PerplexityBot 2 hours ago
unblock sofifa.com unblocked Meta-ExternalAgent 2 hours ago
unblock sofifa.com unblocked Bytespider 2 hours ago
unblock sofifa.com unblocked Amazonbot 2 hours ago
unblock sofifa.com unblocked CCBot 2 hours ago
score bedbathandbeyond.com changed score (86 → 74) 2 hours ago
score reserved.com changed score (53 → 68) 2 hours ago
charge-stop sfchronicle.com stopped charging ClaudeBot 2 hours ago
score garanteprivacy.it changed score (46 → 73) 2 hours ago
edge-serve ilpost.it stopped refusing at the edge ClaudeBot 2 hours ago
edge-serve webnode.com stopped refusing at the edge ClaudeBot 2 hours ago
block emol.com started blocking Googlebot-News 2 hours ago
unblock chavesnamao.com.br unblocked GPTBot 2 hours ago
unblock chavesnamao.com.br unblocked ClaudeBot 2 hours ago
unblock chavesnamao.com.br unblocked Google-Extended 2 hours ago
unblock chavesnamao.com.br unblocked Applebot-Extended 2 hours ago
unblock chavesnamao.com.br unblocked Meta-ExternalAgent 2 hours ago
unblock chavesnamao.com.br unblocked Amazonbot 2 hours ago
unblock chavesnamao.com.br unblocked CCBot 2 hours ago
unblock chavesnamao.com.br unblocked cohere-ai 2 hours ago
edge-refuse thuvienphapluat.vn started refusing at the edge GPTBot 3 hours ago
edge-refuse thuvienphapluat.vn started refusing at the edge OAI-SearchBot 3 hours ago
edge-refuse thuvienphapluat.vn started refusing at the edge ClaudeBot 3 hours ago
score thuvienphapluat.vn changed score (83 → 70) 3 hours ago
score sinsay.com changed score (53 → 67) 3 hours ago
unblock tv2.dk unblocked GPTBot 3 hours ago
unblock tv2.dk unblocked OAI-SearchBot 3 hours ago
unblock tv2.dk unblocked ChatGPT-User 3 hours ago
unblock tv2.dk unblocked ClaudeBot 3 hours ago
unblock tv2.dk unblocked Claude-User 3 hours ago
unblock tv2.dk unblocked Claude-SearchBot 3 hours ago
unblock tv2.dk unblocked anthropic-ai 3 hours ago
unblock tv2.dk unblocked Google-Extended 3 hours ago
unblock tv2.dk unblocked Google-CloudVertexBot 3 hours ago
unblock tv2.dk unblocked GoogleOther 3 hours ago
unblock tv2.dk unblocked Applebot 3 hours ago
unblock tv2.dk unblocked Applebot-Extended 3 hours ago
unblock tv2.dk unblocked PerplexityBot 3 hours ago
unblock tv2.dk unblocked Perplexity-User 3 hours ago
unblock tv2.dk unblocked Meta-ExternalAgent 3 hours ago
unblock tv2.dk unblocked Meta-ExternalFetcher 3 hours ago

Available as RSS and as JSON. Journalists and researchers are welcome to use it with attribution.