Bot Detection

Bot Detection

Websites generally want to differentiate bots from human users and ideally block those that are malicious or not useful. Certain verticals typically require less or more effort when scraping sites. Below we outlined some of our experiences in bullet points. How do sites ideally protect themselves against scraping/crawling? Putting a strict rate limit on data-center…

Is scraping legal?
|

Is scraping legal?

Scraping can be a divisive topic. In general, it is perceived as something slightly shady or illicit by people who are unfamiliar with it. Consequently, the legality of scraping is frequently questioned. This article will clear up some of the common misconceptions and break down the facts on legal scraping. Disclaimer: We are not lawyers,…

What is a Proxy?

What is a Proxy?

The term proxy originates from the Latin word procuro – to manage or administer. It refers to an intermediary or substitute for something. In IT, a proxy is a middleman server that connects the client to the server which can service their request. Instead of a direct connection to a website, a proxy server routes…