{"id":242,"date":"2012-09-12T18:07:41","date_gmt":"2012-09-12T18:07:41","guid":{"rendered":"http:\/\/www.price2spy.com\/blog\/?p=242"},"modified":"2016-06-28T14:26:39","modified_gmt":"2016-06-28T14:26:39","slug":"need-all-competitor-products-crawled-monitored","status":"publish","type":"post","link":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/","title":{"rendered":"Need ALL competitor products crawled \/ monitored?"},"content":{"rendered":"<p><img loading=\"lazy\" decoding=\"async\" class=\"alignleft size-thumbnail wp-image-243\" title=\"Automated site crawl\" src=\"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2012\/09\/web_crawler-150x150.jpg\" alt=\"Automated site crawl\" width=\"150\" height=\"150\" \/>Quite recently Price2Spy was hired to crawl and scrape all product pages from our client&#8217;s competitor website.<\/p>\n<p>The task may sound simple, but there was a catch: there were more than 300K pages to scrape, not all of them were unique (complex content logic involved), and we had a deadline of 1 working week.<\/p>\n<p>The task was tough, but we did make it &#8211; thanks to our experience in site scraping and crawling technologies, as well as our distributed server technology, which enabled us to crawl at a very fast rate, without causing suspicion on target website.<\/p>\n<p>Anyway, what our client got from us was a series of Excel files, with more than 370 000 products, with product details such as<\/p>\n<ul>\n<li>product name<\/li>\n<li>product URL<\/li>\n<li>product description<\/li>\n<li>product category<\/li>\n<li>product price (list \/ sale price)<\/li>\n<li>brand information<\/li>\n<li>stock levels<\/li>\n<li>manufacturer part number (MPN)<\/li>\n<\/ul>\n<p>The next step was on our client, who filtered out the product list (he selected only products that he needed monitored) &#8211; and we have loaded those products into Price2Spy. Today all those products price \/ availability data is checked on daily basis &#8211; which is something of a great value to our client.<\/p>\n<p>Smart, isn&#8217;t it?<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Quite recently Price2Spy was hired to crawl and scrape all product pages from our client&#8217;s competitor website. The task may sound simple, but there was a catch: there were more than 300K pages to scrape, not all of them were unique (complex content logic involved),&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[167],"tags":[81,83],"class_list":["post-242","post","type-post","status-publish","format-standard","hentry","category-new-price2spy-features","tag-product-matching","tag-site-crawl-2"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Need ALL competitor products crawled \/ monitored? - Price2Spy\u00ae Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Need ALL competitor products crawled \/ monitored? - Price2Spy\u00ae Blog\" \/>\n<meta property=\"og:description\" content=\"Quite recently Price2Spy was hired to crawl and scrape all product pages from our client&#8217;s competitor website. The task may sound simple, but there was a catch: there were more than 300K pages to scrape, not all of them were unique (complex content logic involved),...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/\" \/>\n<meta property=\"og:site_name\" content=\"Price2Spy\u00ae Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Price2Spy\/\" \/>\n<meta property=\"article:published_time\" content=\"2012-09-12T18:07:41+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-06-28T14:26:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2012\/09\/web_crawler-150x150.jpg\" \/>\n<meta name=\"author\" content=\"wpt_admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Price2Spy\" \/>\n<meta name=\"twitter:site\" content=\"@Price2Spy\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"wpt_admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Need ALL competitor products crawled \/ monitored? - Price2Spy\u00ae Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/","og_locale":"en_US","og_type":"article","og_title":"Need ALL competitor products crawled \/ monitored? - Price2Spy\u00ae Blog","og_description":"Quite recently Price2Spy was hired to crawl and scrape all product pages from our client&#8217;s competitor website. The task may sound simple, but there was a catch: there were more than 300K pages to scrape, not all of them were unique (complex content logic involved),...","og_url":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/","og_site_name":"Price2Spy\u00ae Blog","article_publisher":"https:\/\/www.facebook.com\/Price2Spy\/","article_published_time":"2012-09-12T18:07:41+00:00","article_modified_time":"2016-06-28T14:26:39+00:00","og_image":[{"url":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2012\/09\/web_crawler-150x150.jpg","type":"","width":"","height":""}],"author":"wpt_admin","twitter_card":"summary_large_image","twitter_creator":"@Price2Spy","twitter_site":"@Price2Spy","twitter_misc":{"Written by":"wpt_admin","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/#article","isPartOf":{"@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/"},"author":{"name":"wpt_admin","@id":"https:\/\/www.price2spy.com\/blog\/#\/schema\/person\/709b1bc4720ca6a59a82eced04fe8796"},"headline":"Need ALL competitor products crawled \/ monitored?","datePublished":"2012-09-12T18:07:41+00:00","dateModified":"2016-06-28T14:26:39+00:00","mainEntityOfPage":{"@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/"},"wordCount":205,"commentCount":0,"image":{"@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/#primaryimage"},"thumbnailUrl":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2012\/09\/web_crawler-150x150.jpg","keywords":["product matching","site crawl"],"articleSection":["New Price2Spy features"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/","url":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/","name":"Need ALL competitor products crawled \/ monitored? - Price2Spy\u00ae Blog","isPartOf":{"@id":"https:\/\/www.price2spy.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/#primaryimage"},"image":{"@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/#primaryimage"},"thumbnailUrl":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2012\/09\/web_crawler-150x150.jpg","datePublished":"2012-09-12T18:07:41+00:00","dateModified":"2016-06-28T14:26:39+00:00","author":{"@id":"https:\/\/www.price2spy.com\/blog\/#\/schema\/person\/709b1bc4720ca6a59a82eced04fe8796"},"breadcrumb":{"@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/#primaryimage","url":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2012\/09\/web_crawler-150x150.jpg","contentUrl":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2012\/09\/web_crawler-150x150.jpg"},{"@type":"BreadcrumbList","@id":"https:\/\/www.price2spy.com\/blog\/need-all-competitor-products-crawled-monitored\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.price2spy.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Need ALL competitor products crawled \/ monitored?"}]},{"@type":"WebSite","@id":"https:\/\/www.price2spy.com\/blog\/#website","url":"https:\/\/www.price2spy.com\/blog\/","name":"Price2Spy\u00ae Blog","description":"Price2Spy\u00ae","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.price2spy.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.price2spy.com\/blog\/#\/schema\/person\/709b1bc4720ca6a59a82eced04fe8796","name":"wpt_admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/8bc6d067b2316466cc3b0f650f3afe7053ea9964796bb7453f6a1feb7d7d79e3?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/8bc6d067b2316466cc3b0f650f3afe7053ea9964796bb7453f6a1feb7d7d79e3?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8bc6d067b2316466cc3b0f650f3afe7053ea9964796bb7453f6a1feb7d7d79e3?s=96&d=mm&r=g","caption":"wpt_admin"},"url":"https:\/\/www.price2spy.com\/blog\/author\/wpt_admin\/"}]}},"_links":{"self":[{"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/posts\/242","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/comments?post=242"}],"version-history":[{"count":1,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/posts\/242\/revisions"}],"predecessor-version":[{"id":3593,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/posts\/242\/revisions\/3593"}],"wp:attachment":[{"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/media?parent=242"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/categories?post=242"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/tags?post=242"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}