Need ALL competitor products crawled / monitored?

New Price2Spy features 12.9.2012. Reading Time: < 1

Automated site crawlQuite recently Price2Spy was hired to crawl and scrape all product pages from our client’s competitor website.

The task may sound simple, but there was a catch: there were more than 300K pages to scrape, not all of them were unique (complex content logic involved), and we had a deadline of 1 working week.

The task was tough, but we did make it – thanks to our experience in site scraping and crawling technologies, as well as our distributed server technology, which enabled us to crawl at a very fast rate, without causing suspicion on target website.

Anyway, what our client got from us was a series of Excel files, with more than 370 000 products, with product details such as

  • product name
  • product URL
  • product description
  • product category
  • product price (list / sale price)
  • brand information
  • stock levels
  • manufacturer part number (MPN)

The next step was on our client, who filtered out the product list (he selected only products that he needed monitored) – and we have loaded those products into Price2Spy. Today all those products price / availability data is checked on daily basis – which is something of a great value to our client.

Smart, isn’t it?