Price2Spy Web Crawler
Data extraction: Web crawling and Web scraping
Need to extract all product data from a website in one go?
Need a detailed overview of what your competitor’s assortment, along with full pricing / product description, category and brand information?
Price2Spy team has mastered a web crawling & web scraping process that could help your business gather such valuable data in bulk.
Just let us know the list of sites where you want the data extracted from – and the rest is on us!
Key Features of Price2Spy Crawler
Some of the key technical strengths that distinguish Price2Spy Crawler from other similar tools:
- Ability to crawl and extract data from very complex websites
- having very complex page navigation structure
- having complex JavaScript menu and/or paging implementation
- having strong anti-bot protection (sites that do not want to be crawled) – for example, Amazon
- capturing multiple product variations shown on the same product page
- requiring browser interaction before scraping data
- having huge amounts of products
- Crawl size / location
- we can crawl websites in any language / any country
- crawling entire website / crawling only specific product categories/brands (for example – it does make much sense in crawling whole of Amazon. However, crawling several specific product categories on Amazon can be done)
- crawling websites that are location-sensitive (showing different results depending on visitor’s IP / ZIP code). For example: Amazon will show different results if you’re an international vs US visitor
- big (more than 1 000 000 product pages) or small (less than 1000 product pages) – can be done
Extraction Results
-
- We have the ability to capture data fields which are not shown on product page itself (for example: fields shown on category page, shown before reaching the product page)
- Extraction results can be delivered in a list (Excel, CSV, XML). If you need a custom format, do let us know
- Extraction results can be ran against automated translation services (so you get results in your preferred language)
- One-off or repetitive
- Price2Spy’s Product Extraction service can be used as one-off data source, or as a repetitive process (in which case you will be able to determine delta’s)
- New products – products which have been added to the website since the last crawl took place
- Deleted products – products which have been removed from the website since the last crawl took place
- If you go for a repetitive crawl / extraction, you will be able to determine the recrawl frequency
- Price2Spy’s Product Extraction service can be used as one-off data source, or as a repetitive process (in which case you will be able to determine delta’s)
- Product matching
- Products that have been extracted can be matched (automatically or manually) to your own products
Use cases
We have performed crawl / extraction operations for a multitude of our clients, and we have noticed that they can be roughly grouped into following use cases
- For online retailers
- Extracting complete competitor’s assortment (to be used as data source for adding new products on own store)
- Extracting delta’s in competitor’s assortment
- Knowing which products have been added to competitor website
- Knowing which products have been discontinued by your competitor
- (of course, this requires a periodical recrawl)
- For Brands / Distributors
- Extracting product reviews from retail websites, in order to determine consumer sentiment towards the product (this service is sometimes combined with automated review translation)
- Extracting newly released products from competitor brands
- Data science – providing product (or user review) data for in-house data science projects. Data can be provided in original language, or machine-translated
- For Marketing Agencies
- Extracting products from Ecommerce sites (online stores) on behalf of it’s clients
- Extracting products and their reviews from Review websites (for example: capterra.com, g2crowd.com)
- Data science – providing product (or user review) data for data science projects performed on behalf of agency clients. Data can be provided in original language, or machine-translated
Assortment monitor
Assortment Monitor is a feature designed to suggest products from any website of your choice – products that you do not monitor, but could consider monitoring them.
Assortment Monitor offers basic product data at your disposal completely free of charge (product name, Brand, URL) coming from various sources like proactive site crawls (performed on Price2Spy’s initiative), public data feeds, price comparison websites etc.
However, if you need a more detailed product crawl (with all product fields like Description, ImageURL, NumberOfReviews, AverageRating etc etc) please contact support@price2spy.com – we will be happy to give you a quote.
All Price2Spy services at a glance
Here you can find an overview of all the services Price2Spy offers – hopefully it will help you understand what can be achieved,
and the best way Price2Spy services can fit your business and it’s needs.