Amazon Data Scraper

Case Study

With the fast rise of online retail, online marketplace marketing is at the forefront of any digital marketing agency. Modern software solutions can automate the marketplace marketing efforts and aid in decision-making, which is why our client, a major marketing agency, approached us with a task of developing a data scraping solution for Amazon.

The data scraper would have to navigate the Amazon website, collect data while filtering it according to the client requirements, export and sort it according to the client's requirements.

Scraping Data From Amazon

The scraper goes through Amazon shops simultaneously on the US, UK and German websites. The scraper collects information on the product categories present on Amazon, including all subcategories, which are temporarily uploaded into a database.

Next, the scraper looks at each product category and extracts product and shop information. The information about the products is cached to avoid duplicates. The shop information is then temporarily extracted into a database as well.

Lastly, each shop and its products are analyzed to check if it fits the client's requirements. Our scraper checks against the following criteria: shop owner, lowest and highest product price, product types, average feedback, average number of reviews. The shops that don't fit the desired characteristics are removed from a database, leaving only the shops that do.

The database can be accessed directly or uploaded into a CMS for ease of use.

CAPTCHA and Avoiding Blocks

We have developed our scraper to be very hard to detect by Amazon. The scraper effectively avoids being blocked in multiple ways:

• emulation of human behavior, like handling pagination and performing scrolling;
• rotation of user agents;
• dynamic IP changes.

The scraper can also change the data scraping speed, slowing down to avoid suspicion.

We have also developed a CAPTCHA solving module, making our data scraping solution fully autonomous.

Results

Our data scraper is a full-fledged solution for scraping and filtering large volumes of data from different Amazon websites. The scraper has been successfully launched and helps our client achieve their marketing goals.