Websites generally want to differentiate bots from human users and ideally block those that are malicious or not useful. Certain verticals typically require less or more effort when scraping sites. Below we outlined some of our experiences in bullet points. How do sites ideally protect themselves against scraping/crawling? Putting a strict rate limit on data-center…