Web Crawling Case Study: Using Web Crawler in E-commerce

The number of e-commerce companies which use web crawler is growing rapidly due to the current competitive market conditions.

There are many industry examples where Price2Spy web crawler is currently being used by many of our clients.

However, since the use purpose of web crawler varies on a case to case basis, we thought that the following web crawling case study might help you to better understand how Price2Spy web crawler works and how your business might benefit from it.

Click here to see more web crawling case studies from Price2Spy.

Price2Spy Web Crawler helps Australia’s leading retailers to crawl data from the world’s technically most challenging Websites

This client is one of Australia’s leading retailers of a broad range of products. With more than 50 years of experience on the market, 186 stores and more than 20.000 members across the world, they are operating in the ever-changing market with great success.

When the client approached us, their business was confronted with highly competitive market where their competitors frequently changed their prices and assortment, which was very difficult to track, considering the large scope of products to be monitored across very different product categories. In other words;

1. The client needed to analyze their competitors’ product assortment in order to expand their own assortment and stay competitive.
2. The client was in need of reliable information on competitors’ actions regarding how they run campaigns and promotions.

It was clear that this wasn’t a simple task. But, what made the whole process even more challenging was the complex site structure of the competitor sites and a large number of products.

On top of that, we needed to perform web crawling on world’s technically most challenging websites – bot-aware sites, CAPTCHA protected and location-sensitive (showing different results depending on visitor’s IP / ZIP code) sites.

In order for our client to stay on top of their competitors, we knew that we need to use our advanced web crawler Price2Spy’s Web Crawler is a non-standard advanced service used by our clients to collect all required data from particular websites that is later being used to manage an in-house product database.

At the end of this process, what our client got from us was a series of Excel files, with more than 500 000 products, including  product details such as;

Our client performed the next steps – they filtered out the product list and selected only products that they wanted to focus on. We have then loaded those products into Price2Spy where we are checking their price/availability data on a daily basis – allowing our client to get wholesome and up to date data.

If you also need to gather data from a retailer or competitor site in one go, or if you need a detailed overview of your competitor’s assortment, along with full pricing/product description, ratings and reviews, category and brand information and etc., just send us the list of websites you want to be scraped, and a list of information you expect to get, and we’ll be happy to help you out with this task.