Change log

Who changed their mind

Crawler policy is edited quietly, usually with no announcement. Every scan is compared against the previous one for that domain, and any difference is recorded here permanently.

300
recorded changes
104
crawlers newly blocked
134
crawlers newly allowed

Every recorded change newest first

  • edge-serve gm.com stopped refusing at the edge GPTBot 1 day ago
  • edge-serve gm.com stopped refusing at the edge OAI-SearchBot 1 day ago
  • edge-serve gm.com stopped refusing at the edge ClaudeBot 1 day ago
  • edge-serve gm.com stopped refusing at the edge PerplexityBot 1 day ago
  • edge-refuse shopee.co.th started refusing at the edge ClaudeBot 1 day ago
  • unblock amazon.sa unblocked GPTBot 1 day ago
  • unblock amazon.sa unblocked OAI-SearchBot 1 day ago
  • unblock amazon.sa unblocked ChatGPT-User 1 day ago
  • unblock amazon.sa unblocked ClaudeBot 1 day ago
  • unblock amazon.sa unblocked Claude-User 1 day ago
  • unblock amazon.sa unblocked Claude-SearchBot 1 day ago
  • unblock amazon.sa unblocked Google-Extended 1 day ago
  • unblock amazon.sa unblocked Google-CloudVertexBot 1 day ago
  • unblock amazon.sa unblocked GoogleOther 1 day ago
  • unblock amazon.sa unblocked PerplexityBot 1 day ago
  • unblock amazon.sa unblocked Perplexity-User 1 day ago
  • unblock amazon.sa unblocked Meta-ExternalAgent 1 day ago
  • unblock amazon.sa unblocked Meta-ExternalFetcher 1 day ago
  • unblock amazon.sa unblocked Meta-WebIndexer 1 day ago
  • unblock amazon.sa unblocked Bytespider 1 day ago
  • unblock amazon.sa unblocked CCBot 1 day ago
  • unblock amazon.sa unblocked Diffbot 1 day ago
  • unblock amazon.sa unblocked omgili 1 day ago
  • unblock amazon.sa unblocked omgilibot 1 day ago
  • unblock amazon.sa unblocked AI2Bot 1 day ago
  • unblock amazon.sa unblocked cohere-ai 1 day ago
  • unblock amazon.sa unblocked cohere-training-data-crawler 1 day ago
  • unblock amazon.sa unblocked MistralAI-User 1 day ago
  • unblock amazon.sa unblocked DuckAssistBot 1 day ago
  • unblock amazon.sa unblocked YouBot 1 day ago
  • unblock amazon.sa unblocked PanguBot 1 day ago
  • unblock amazon.sa unblocked Timpibot 1 day ago
  • unblock amazon.sa unblocked ImagesiftBot 1 day ago
  • unblock amazon.sa unblocked Kangaroo Bot 1 day ago
  • unblock amazon.sa unblocked Scrapy 1 day ago
  • llms added kia.com published an llms.txt 1 day ago
  • score carfax.com changed score (59 → 89) 1 day ago
  • edge-refuse apachelounge.com started refusing at the edge GPTBot 1 day ago
  • edge-refuse apachelounge.com started refusing at the edge OAI-SearchBot 1 day ago
  • llms added buymeacoffee.com published an llms.txt 1 day ago
  • edge-refuse abril.com.br started refusing at the edge OAI-SearchBot 2 days ago
  • edge-refuse emojipedia.org started refusing at the edge PerplexityBot 2 days ago
  • edge-refuse steaminventoryhelper.com started refusing at the edge GPTBot 2 days ago
  • edge-refuse steaminventoryhelper.com started refusing at the edge OAI-SearchBot 2 days ago
  • edge-refuse steaminventoryhelper.com started refusing at the edge ClaudeBot 2 days ago
  • edge-refuse steaminventoryhelper.com started refusing at the edge PerplexityBot 2 days ago
  • score steaminventoryhelper.com changed score (61 → 47) 2 days ago
  • block networkofcare.org started blocking GPTBot 3 days ago
  • block networkofcare.org started blocking ClaudeBot 3 days ago
  • block networkofcare.org started blocking anthropic-ai 3 days ago
  • block networkofcare.org started blocking Google-Extended 3 days ago
  • block networkofcare.org started blocking Google-CloudVertexBot 3 days ago
  • block networkofcare.org started blocking GoogleOther 3 days ago
  • block networkofcare.org started blocking Applebot-Extended 3 days ago
  • block networkofcare.org started blocking Meta-ExternalAgent 3 days ago
  • block networkofcare.org started blocking Bytespider 3 days ago
  • block networkofcare.org started blocking Amazonbot 3 days ago
  • block networkofcare.org started blocking CCBot 3 days ago
  • block networkofcare.org started blocking Diffbot 3 days ago
  • block networkofcare.org started blocking omgili 3 days ago
  • block networkofcare.org started blocking omgilibot 3 days ago
  • block networkofcare.org started blocking cohere-ai 3 days ago
  • block networkofcare.org started blocking MistralAI-Training 3 days ago
  • score cellrebel.com changed score (69 → 46) 3 days ago
  • edge-serve mainichi.jp stopped refusing at the edge GPTBot 3 days ago
  • edge-serve mainichi.jp stopped refusing at the edge OAI-SearchBot 3 days ago
  • edge-serve mainichi.jp stopped refusing at the edge ClaudeBot 3 days ago
  • edge-serve mainichi.jp stopped refusing at the edge PerplexityBot 3 days ago
  • score pchome.com.tw changed score (84 → 52) 3 days ago
  • block sbs.com.au started blocking FacebookBot 3 days ago
  • block howtogeek.com started blocking Timpibot 3 days ago
  • block howtogeek.com started blocking ImagesiftBot 3 days ago
  • unblock thestar.com unblocked GPTBot 3 days ago
  • edge-serve thestar.com stopped refusing at the edge GPTBot 3 days ago
  • unblock thestar.com unblocked OAI-SearchBot 3 days ago
  • edge-serve thestar.com stopped refusing at the edge OAI-SearchBot 3 days ago
  • unblock thestar.com unblocked ChatGPT-User 3 days ago
  • unblock thestar.com unblocked ClaudeBot 3 days ago
  • edge-serve thestar.com stopped refusing at the edge ClaudeBot 3 days ago
  • unblock thestar.com unblocked Claude-User 3 days ago
  • unblock thestar.com unblocked Claude-SearchBot 3 days ago
  • unblock thestar.com unblocked anthropic-ai 3 days ago
  • unblock thestar.com unblocked Google-Extended 3 days ago
  • unblock thestar.com unblocked Google-CloudVertexBot 3 days ago
  • unblock thestar.com unblocked GoogleOther 3 days ago
  • unblock thestar.com unblocked Applebot-Extended 3 days ago
  • unblock thestar.com unblocked PerplexityBot 3 days ago
  • edge-serve thestar.com stopped refusing at the edge PerplexityBot 3 days ago
  • unblock thestar.com unblocked Perplexity-User 3 days ago
  • unblock thestar.com unblocked Meta-ExternalAgent 3 days ago
  • unblock thestar.com unblocked Meta-ExternalFetcher 3 days ago
  • unblock thestar.com unblocked FacebookBot 3 days ago
  • unblock thestar.com unblocked Meta-WebIndexer 3 days ago
  • unblock thestar.com unblocked Bytespider 3 days ago
  • unblock thestar.com unblocked Amazonbot 3 days ago
  • unblock thestar.com unblocked Amzn-SearchBot 3 days ago
  • unblock thestar.com unblocked Amzn-User 3 days ago
  • unblock thestar.com unblocked CCBot 3 days ago
  • unblock thestar.com unblocked Diffbot 3 days ago
  • unblock thestar.com unblocked omgili 3 days ago
  • unblock thestar.com unblocked omgilibot 3 days ago
  • unblock thestar.com unblocked AI2Bot 3 days ago
  • unblock thestar.com unblocked cohere-ai 3 days ago
  • unblock thestar.com unblocked cohere-training-data-crawler 3 days ago
  • unblock thestar.com unblocked MistralAI-User 3 days ago
  • unblock thestar.com unblocked MistralAI-Index 3 days ago
  • unblock thestar.com unblocked DuckAssistBot 3 days ago
  • unblock thestar.com unblocked YouBot 3 days ago
  • unblock thestar.com unblocked PanguBot 3 days ago
  • unblock thestar.com unblocked Timpibot 3 days ago
  • unblock thestar.com unblocked ImagesiftBot 3 days ago
  • unblock thestar.com unblocked Kangaroo Bot 3 days ago
  • unblock thestar.com unblocked SemrushBot-OCOB 3 days ago
  • unblock thestar.com unblocked Scrapy 3 days ago
  • score thestar.com changed score (64 → 46) 3 days ago
  • unblock amazon.pl unblocked GPTBot 3 days ago
  • unblock amazon.pl unblocked OAI-SearchBot 3 days ago
  • unblock amazon.pl unblocked ChatGPT-User 3 days ago
  • unblock amazon.pl unblocked ClaudeBot 3 days ago
  • unblock amazon.pl unblocked Claude-User 3 days ago
  • unblock amazon.pl unblocked Claude-SearchBot 3 days ago
  • unblock amazon.pl unblocked Google-Extended 3 days ago
  • unblock amazon.pl unblocked Google-CloudVertexBot 3 days ago
  • unblock amazon.pl unblocked GoogleOther 3 days ago
  • unblock amazon.pl unblocked PerplexityBot 3 days ago
  • unblock amazon.pl unblocked Perplexity-User 3 days ago
  • unblock amazon.pl unblocked Meta-ExternalAgent 3 days ago
  • unblock amazon.pl unblocked Meta-ExternalFetcher 3 days ago
  • unblock amazon.pl unblocked Meta-WebIndexer 3 days ago
  • unblock amazon.pl unblocked Bytespider 3 days ago
  • unblock amazon.pl unblocked CCBot 3 days ago
  • unblock amazon.pl unblocked Diffbot 3 days ago
  • unblock amazon.pl unblocked omgili 3 days ago
  • unblock amazon.pl unblocked omgilibot 3 days ago
  • unblock amazon.pl unblocked AI2Bot 3 days ago
  • unblock amazon.pl unblocked cohere-ai 3 days ago
  • unblock amazon.pl unblocked cohere-training-data-crawler 3 days ago
  • unblock amazon.pl unblocked MistralAI-User 3 days ago
  • unblock amazon.pl unblocked DuckAssistBot 3 days ago
  • unblock amazon.pl unblocked YouBot 3 days ago
  • unblock amazon.pl unblocked PanguBot 3 days ago
  • unblock amazon.pl unblocked Timpibot 3 days ago
  • unblock amazon.pl unblocked ImagesiftBot 3 days ago
  • unblock amazon.pl unblocked Kangaroo Bot 3 days ago
  • unblock amazon.pl unblocked Scrapy 3 days ago
  • block uiowa.edu started blocking GPTBot 3 days ago
  • edge-refuse uiowa.edu started refusing at the edge GPTBot 3 days ago
  • block uiowa.edu started blocking ClaudeBot 3 days ago
  • edge-refuse uiowa.edu started refusing at the edge ClaudeBot 3 days ago
  • block uiowa.edu started blocking anthropic-ai 3 days ago
  • block uiowa.edu started blocking Google-Extended 3 days ago
  • block uiowa.edu started blocking Google-CloudVertexBot 3 days ago
  • block uiowa.edu started blocking GoogleOther 3 days ago
  • block uiowa.edu started blocking Applebot-Extended 3 days ago
  • edge-refuse uiowa.edu started refusing at the edge PerplexityBot 3 days ago
  • block uiowa.edu started blocking Meta-ExternalAgent 3 days ago
  • block uiowa.edu started blocking Bytespider 3 days ago
  • block uiowa.edu started blocking Amazonbot 3 days ago
  • block uiowa.edu started blocking CCBot 3 days ago
  • block uiowa.edu started blocking Diffbot 3 days ago
  • block uiowa.edu started blocking omgili 3 days ago
  • block uiowa.edu started blocking omgilibot 3 days ago
  • block uiowa.edu started blocking cohere-ai 3 days ago
  • block uiowa.edu started blocking MistralAI-Training 3 days ago
  • score uiowa.edu changed score (86 → 70) 3 days ago
  • edge-refuse vaticannews.va started refusing at the edge PerplexityBot 3 days ago
  • block screenrant.com started blocking Timpibot 3 days ago
  • block screenrant.com started blocking ImagesiftBot 3 days ago
  • block seekingalpha.com started blocking GPTBot 3 days ago
  • unblock seekingalpha.com unblocked ChatGPT-User 3 days ago
  • unblock seekingalpha.com unblocked Claude-User 3 days ago
  • block seekingalpha.com started blocking Applebot-Extended 3 days ago
  • unblock seekingalpha.com unblocked Perplexity-User 3 days ago
  • unblock seekingalpha.com unblocked Meta-ExternalFetcher 3 days ago
  • block seekingalpha.com started blocking Amazonbot 3 days ago
  • unblock seekingalpha.com unblocked MistralAI-User 3 days ago
  • unblock seekingalpha.com unblocked DuckAssistBot 3 days ago
  • unblock seekingalpha.com unblocked YouBot 3 days ago
  • score vecteezy.com changed score (87 → 49) 3 days ago
  • edge-refuse readthedocs.org started refusing at the edge PerplexityBot 3 days ago
  • block alltrails.com started blocking SemrushBot-OCOB 3 days ago
  • edge-refuse routledge.com started refusing at the edge PerplexityBot 3 days ago
  • block menshealth.com started blocking anthropic-ai 3 days ago
  • block menshealth.com started blocking Google-Extended 3 days ago
  • block menshealth.com started blocking Bytespider 3 days ago
  • block menshealth.com started blocking Amazonbot 3 days ago
  • block menshealth.com started blocking CCBot 3 days ago
  • block rte.ie started blocking ClaudeBot 3 days ago
  • block rte.ie started blocking Applebot-Extended 3 days ago
  • block rte.ie started blocking Meta-ExternalAgent 3 days ago
  • block rte.ie started blocking Bytespider 3 days ago
  • block rte.ie started blocking Amazonbot 3 days ago
  • block rte.ie started blocking cohere-training-data-crawler 3 days ago
  • edge-serve freelancer.com stopped refusing at the edge PerplexityBot 3 days ago
  • unblock radiofrance.fr unblocked GPTBot 3 days ago
  • unblock radiofrance.fr unblocked OAI-SearchBot 3 days ago
  • unblock radiofrance.fr unblocked ChatGPT-User 3 days ago
  • unblock radiofrance.fr unblocked ClaudeBot 3 days ago
  • unblock radiofrance.fr unblocked anthropic-ai 3 days ago
  • unblock radiofrance.fr unblocked Google-Extended 3 days ago
  • unblock radiofrance.fr unblocked Google-CloudVertexBot 3 days ago
  • unblock radiofrance.fr unblocked Applebot-Extended 3 days ago
  • unblock radiofrance.fr unblocked PerplexityBot 3 days ago
  • unblock radiofrance.fr unblocked Meta-ExternalAgent 3 days ago
  • unblock radiofrance.fr unblocked Meta-ExternalFetcher 3 days ago
  • unblock radiofrance.fr unblocked FacebookBot 3 days ago
  • unblock radiofrance.fr unblocked Meta-WebIndexer 3 days ago
  • unblock radiofrance.fr unblocked Bytespider 3 days ago
  • unblock radiofrance.fr unblocked TikTokSpider 3 days ago
  • unblock radiofrance.fr unblocked Amazonbot 3 days ago
  • unblock radiofrance.fr unblocked CCBot 3 days ago
  • unblock radiofrance.fr unblocked omgili 3 days ago
  • unblock radiofrance.fr unblocked omgilibot 3 days ago
  • unblock radiofrance.fr unblocked AI2Bot 3 days ago
  • unblock radiofrance.fr unblocked cohere-ai 3 days ago
  • unblock radiofrance.fr unblocked cohere-training-data-crawler 3 days ago
  • unblock radiofrance.fr unblocked MistralAI-User 3 days ago
  • unblock radiofrance.fr unblocked DuckAssistBot 3 days ago
  • unblock radiofrance.fr unblocked PanguBot 3 days ago
  • unblock radiofrance.fr unblocked Timpibot 3 days ago
  • unblock radiofrance.fr unblocked ImagesiftBot 3 days ago
  • unblock radiofrance.fr unblocked Scrapy 3 days ago
  • score irancell.ir changed score (49 → 89) 3 days ago
  • llms added zenodo.org published an llms.txt 3 days ago
  • unblock pcworld.com unblocked Google-Extended 3 days ago
  • score legislation.gov.uk changed score (72 → 47) 3 days ago
  • score securityfocus.com changed score (46 → 66) 3 days ago
  • edge-serve shopee.vn stopped refusing at the edge ClaudeBot 3 days ago
  • llms added aruba.it published an llms.txt 3 days ago
  • score polyspeak.ai changed score (87 → 56) 3 days ago
  • score consentmanager.net changed score (46 → 57) 3 days ago
  • edge-serve cian.ru stopped refusing at the edge GPTBot 3 days ago
  • edge-serve cian.ru stopped refusing at the edge OAI-SearchBot 3 days ago
  • edge-serve cian.ru stopped refusing at the edge ClaudeBot 3 days ago
  • block oregonstate.edu started blocking GPTBot 3 days ago
  • block oregonstate.edu started blocking ClaudeBot 3 days ago
  • block oregonstate.edu started blocking anthropic-ai 3 days ago
  • block oregonstate.edu started blocking Google-Extended 3 days ago
  • block oregonstate.edu started blocking Google-CloudVertexBot 3 days ago
  • block oregonstate.edu started blocking GoogleOther 3 days ago
  • block oregonstate.edu started blocking Applebot-Extended 3 days ago
  • edge-refuse oregonstate.edu started refusing at the edge PerplexityBot 3 days ago
  • block oregonstate.edu started blocking Meta-ExternalAgent 3 days ago
  • block oregonstate.edu started blocking Bytespider 3 days ago
  • block oregonstate.edu started blocking Amazonbot 3 days ago
  • block oregonstate.edu started blocking CCBot 3 days ago
  • block oregonstate.edu started blocking Diffbot 3 days ago
  • block oregonstate.edu started blocking omgili 3 days ago
  • block oregonstate.edu started blocking omgilibot 3 days ago
  • block oregonstate.edu started blocking cohere-ai 3 days ago
  • block oregonstate.edu started blocking MistralAI-Training 3 days ago
  • edge-serve lonelyplanet.com stopped refusing at the edge GPTBot 3 days ago
  • edge-serve 5movierulz.limited stopped refusing at the edge GPTBot 3 days ago
  • edge-serve 5movierulz.limited stopped refusing at the edge ClaudeBot 3 days ago
  • block xnhau.bio started blocking GPTBot 3 days ago
  • block xnhau.bio started blocking ClaudeBot 3 days ago
  • block xnhau.bio started blocking anthropic-ai 3 days ago
  • block xnhau.bio started blocking Google-Extended 3 days ago
  • block xnhau.bio started blocking Google-CloudVertexBot 3 days ago
  • block xnhau.bio started blocking GoogleOther 3 days ago
  • block xnhau.bio started blocking Applebot-Extended 3 days ago
  • block xnhau.bio started blocking Meta-ExternalAgent 3 days ago
  • block xnhau.bio started blocking Bytespider 3 days ago
  • block xnhau.bio started blocking Amazonbot 3 days ago
  • block xnhau.bio started blocking CCBot 3 days ago
  • block xnhau.bio started blocking Diffbot 3 days ago
  • block xnhau.bio started blocking omgili 3 days ago
  • block xnhau.bio started blocking omgilibot 3 days ago
  • block xnhau.bio started blocking cohere-ai 3 days ago
  • block xnhau.bio started blocking MistralAI-Training 3 days ago
  • edge-refuse filmyzilla46.com started refusing at the edge PerplexityBot 3 days ago
  • llms added blacktoon416.com published an llms.txt 3 days ago
  • edge-serve adscale.de stopped refusing at the edge GPTBot 3 days ago
  • edge-serve adscale.de stopped refusing at the edge ClaudeBot 3 days ago
  • block xnhau.xxx started blocking GPTBot 3 days ago
  • block xnhau.xxx started blocking ClaudeBot 3 days ago
  • block xnhau.xxx started blocking anthropic-ai 3 days ago
  • block xnhau.xxx started blocking Google-Extended 3 days ago
  • block xnhau.xxx started blocking Google-CloudVertexBot 3 days ago
  • block xnhau.xxx started blocking GoogleOther 3 days ago
  • block xnhau.xxx started blocking Applebot-Extended 3 days ago
  • block xnhau.xxx started blocking Meta-ExternalAgent 3 days ago
  • block xnhau.xxx started blocking Bytespider 3 days ago
  • block xnhau.xxx started blocking Amazonbot 3 days ago
  • block xnhau.xxx started blocking CCBot 3 days ago
  • block xnhau.xxx started blocking Diffbot 3 days ago
  • block xnhau.xxx started blocking omgili 3 days ago
  • block xnhau.xxx started blocking omgilibot 3 days ago
  • block xnhau.xxx started blocking cohere-ai 3 days ago
  • block xnhau.xxx started blocking MistralAI-Training 3 days ago
  • edge-serve 5movierulz.support stopped refusing at the edge GPTBot 3 days ago
  • edge-serve 5movierulz.support stopped refusing at the edge ClaudeBot 3 days ago
  • edge-refuse moviezwap.land started refusing at the edge PerplexityBot 3 days ago
  • edge-serve avg.com stopped refusing at the edge ClaudeBot 3 days ago
  • edge-serve avg.com stopped refusing at the edge PerplexityBot 3 days ago
  • score sba.gov changed score (92 → 46) 3 days ago
  • block xnhau.you started blocking GPTBot 3 days ago
  • block xnhau.you started blocking ClaudeBot 3 days ago
  • block xnhau.you started blocking anthropic-ai 3 days ago
  • block xnhau.you started blocking Google-Extended 3 days ago

Available as RSS and as JSON. Journalists and researchers are welcome to use it with attribution.