{"id":7300,"date":"2020-06-19T11:53:02","date_gmt":"2020-06-19T11:53:02","guid":{"rendered":"https:\/\/www.price2spy.com\/blog\/?p=7300"},"modified":"2020-07-28T10:59:28","modified_gmt":"2020-07-28T10:59:28","slug":"part-6-evaluating-ml-training-results","status":"publish","type":"post","link":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/","title":{"rendered":"(Part #6) Evaluating ML training results"},"content":{"rendered":"\n<p> <a rel=\"noreferrer noopener\" aria-label=\"Product matching in Price2Spy  (opens in a new tab)\" href=\"https:\/\/www.price2spy.com\/en\/pricing\/product-matching.html\" target=\"_blank\">Product matching in Price2Spy <\/a><\/p>\n\n\n\n<p><strong> Previous topic:<\/strong>&nbsp; <a rel=\"noreferrer noopener\" aria-label=\"(Part #5) ML training Implementation  (opens in a new tab)\" href=\"https:\/\/www.price2spy.com\/blog\/part-5-ml-training-implementation\/\" target=\"_blank\">(Part #5) ML training Implementation <\/a><\/p>\n\n\n\n<p><strong>Next topic:<\/strong> <a href=\"https:\/\/www.price2spy.com\/blog\/part-7-product-matching-via-ml-post-processing\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\"(Part #7)  Post-processing  (opens in a new tab)\">(Part #7)  Post-processing <\/a><\/p>\n\n\n\n<p>How do you know if your ML model works fine? The answer is seemingly simple: you run all of the potential matching combinations through Random Forest (RF), in order to get a matching score of each combination. Then, if the matching score is above X (matching threshold), you consider it a match \u2013 otherwise you consider it a non-match. Since we\u2019re talking about the training set \u2013 the correct matches have already been established \u2013 we should know the correct answer (whether it\u2019s a good match or not). The more correct answers your model has, the better its accuracy.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Matching accuracy VS Matching sensitivity<\/h3>\n\n\n\n<p>All of the above was easier said than done. Soon enough, we had faced a serious of unexpected questions in our <a href=\"https:\/\/www.price2spy.com\/en\/pricing\/product-matching.html\">product matching project<\/a><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>How do we know the ideal value for X (matching score threshold) \u2013 should it be 0.5, 0.9, 0.95 or 0.99?<\/li><li>Matching accuracy is not enough. We need to watch for another dimension, and that is <strong>matching sensitivity<\/strong><\/li><li>a. <strong>Matching accuracy<\/strong> is simple to understand \u2013 if an ML model says that a particular combination of products is a match, matching accuracy means how reliable that piece of information is<\/li><li>b. <strong>Matching sensitivity<\/strong> is more difficult \u2013 it tells us how many matches have been found by an ML model. For example, it may have found a small number of matches extremely accurately \u2013 but that\u2019s only a small number \u2013 we need all matches discovered<\/li><\/ul>\n\n\n\n<p>Please check out the following chart showing how our matching accuracy\/sensitivity evolved, based on the value of X (matching threshold). I must say that from day 1 Accuracy was doing great \u2013 it was a sensitivity that made us work so hard.<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"800\" height=\"600\" src=\"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6.png\" alt=\"ML matching accuracy\/sensitivity\" class=\"wp-image-7301\" srcset=\"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6.png 800w, https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6-768x576.png 768w\" sizes=\"auto, (max-width: 800px) 100vw, 800px\" \/><\/figure><\/div>\n\n\n\n<p>One way of evaluating your model is finding the spot where accuracy and sensitivity curves meet. The higher X for that meeting point \u2013 the better. In our early days, X was 74, as we were improving the features and the post-processing methods, it went up to 85, while now being close to 95.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">ML filling in gaps left by humans<\/h3>\n\n\n\n<p>Pretty soon in our evaluation process we noticed something odd. We had combinations with great matching score (close to 100), which were marked as wrong matches (they were not marked as matches in our training set). When <a href=\"https:\/\/www.price2spy.com\/\">Price2Spy\u2019s<\/a> manual product matching team checked these combinations manually, they were perfectly good. Alarm bell!!! What is wrong with our process?<\/p>\n\n\n\n<p>The answer was not that obvious, but was rather astounding \u2013 ML model has found matches that humans have not! How is that possible? After giving it a lot of thinking, we have come up with several explanations :<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>At the time when manual matching was done &#8211; there was no matching product on Website B. Now there was (it was added in the meantime)<\/li><li>Human matching was not done for the full scope of products (for example, it was done only for certain categories)<\/li><li>In rare cases, there were matches which were omitted by humans<\/li><\/ul>\n\n\n\n<p>OK, this was really great \u2013 ML is finding something humans could not find \u2013 but how do we distinguish which cases are good matches, and which are not? The answer was tough \u2013 the check had to be done manually.<\/p>\n\n\n\n<p>This where our manual matching team came in (thank you, Sandra) &#8211; they did the hard work, so we knew which newly-discovered matches were good, so we could have a better picture of real accuracy\/sensitivity.<\/p>\n\n\n\n<p>And not only that: the fact that ML discovered some matches that were not in our original training set meant that our original training set was not perfect. In that case \u2013 why not add newly discovered matches to the training set, and repeat the process?<\/p>\n\n\n\n<p>This is what we called \u2018iterative ML process\u2019. It took us 4 iterations to reach satisfactory results. Fortunately enough, each new iteration was easier than the previous one, as there were fewer and fewer newly-discovered matches found.<\/p>\n\n\n\n<p><strong>For more information please visit the following links:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><a rel=\"noreferrer noopener\" aria-label=\"Product matching in Price2Spy (opens in a new tab)\" href=\"https:\/\/www.price2spy.com\/en\/pricing\/product-matching.html\" target=\"_blank\">Product matching in Price2Spy<\/a> <\/li><li>Previous topic:  <a rel=\"noreferrer noopener\" aria-label=\"(Part #5) ML training Implementation (opens in a new tab)\" href=\"https:\/\/www.price2spy.com\/blog\/part-5-ml-training-implementation\/\" target=\"_blank\">(Part #5) ML training Implementation<\/a> <\/li><li>Next topic:  <a href=\"https:\/\/www.price2spy.com\/blog\/part-7-product-matching-via-ml-post-processing\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\"(Part #7)  Post-processing  (opens in a new tab)\">(Part #7)  Post-processing <\/a> <\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Product matching in Price2Spy Previous topic:&nbsp; (Part #5) ML training Implementation Next topic: (Part #7) Post-processing How do you know if your ML model works fine? The answer is seemingly simple: you run all of the potential matching combinations through Random Forest (RF), in order&#8230;<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[108,167],"tags":[190,645,646,15,81],"class_list":["post-7300","post","type-post","status-publish","format-standard","hentry","category-best-practices","category-new-price2spy-features","tag-ecommerce","tag-machine-learning","tag-ml","tag-price2spy","tag-product-matching"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>(Part #6) Evaluating ML training results<\/title>\n<meta name=\"description\" content=\"Take a look at how do we evaluate ML training results\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"(Part #6) Evaluating ML training results\" \/>\n<meta property=\"og:description\" content=\"Take a look at how do we evaluate ML training results\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/\" \/>\n<meta property=\"og:site_name\" content=\"Price2Spy\u00ae Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/Price2Spy\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-06-19T11:53:02+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2020-07-28T10:59:28+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6.png\" \/>\n<meta name=\"author\" content=\"Mi\u0161a Kruni\u0107\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@Price2Spy\" \/>\n<meta name=\"twitter:site\" content=\"@Price2Spy\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Mi\u0161a Kruni\u0107\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"(Part #6) Evaluating ML training results","description":"Take a look at how do we evaluate ML training results","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/","og_locale":"en_US","og_type":"article","og_title":"(Part #6) Evaluating ML training results","og_description":"Take a look at how do we evaluate ML training results","og_url":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/","og_site_name":"Price2Spy\u00ae Blog","article_publisher":"https:\/\/www.facebook.com\/Price2Spy\/","article_published_time":"2020-06-19T11:53:02+00:00","article_modified_time":"2020-07-28T10:59:28+00:00","og_image":[{"url":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6.png","type":"","width":"","height":""}],"author":"Mi\u0161a Kruni\u0107","twitter_card":"summary_large_image","twitter_creator":"@Price2Spy","twitter_site":"@Price2Spy","twitter_misc":{"Written by":"Mi\u0161a Kruni\u0107","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/#article","isPartOf":{"@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/"},"author":{"name":"Mi\u0161a Kruni\u0107","@id":"https:\/\/www.price2spy.com\/blog\/#\/schema\/person\/08e388ab2e43e97b3618363fbbe94ded"},"headline":"(Part #6) Evaluating ML training results","datePublished":"2020-06-19T11:53:02+00:00","dateModified":"2020-07-28T10:59:28+00:00","mainEntityOfPage":{"@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/"},"wordCount":685,"commentCount":0,"image":{"@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/#primaryimage"},"thumbnailUrl":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6.png","keywords":["ecommerce","machine learning","ml","price2spy","product matching"],"articleSection":["Best practices in price monitoring","New Price2Spy features"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/","url":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/","name":"(Part #6) Evaluating ML training results","isPartOf":{"@id":"https:\/\/www.price2spy.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/#primaryimage"},"image":{"@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/#primaryimage"},"thumbnailUrl":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6.png","datePublished":"2020-06-19T11:53:02+00:00","dateModified":"2020-07-28T10:59:28+00:00","author":{"@id":"https:\/\/www.price2spy.com\/blog\/#\/schema\/person\/08e388ab2e43e97b3618363fbbe94ded"},"description":"Take a look at how do we evaluate ML training results","breadcrumb":{"@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/#primaryimage","url":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6.png","contentUrl":"https:\/\/www.price2spy.com\/blog\/wp-content\/uploads\/2020\/06\/6.png","width":800,"height":600},{"@type":"BreadcrumbList","@id":"https:\/\/www.price2spy.com\/blog\/part-6-evaluating-ml-training-results\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.price2spy.com\/blog\/"},{"@type":"ListItem","position":2,"name":"(Part #6) Evaluating ML training results"}]},{"@type":"WebSite","@id":"https:\/\/www.price2spy.com\/blog\/#website","url":"https:\/\/www.price2spy.com\/blog\/","name":"Price2Spy\u00ae Blog","description":"Price2Spy\u00ae","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.price2spy.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.price2spy.com\/blog\/#\/schema\/person\/08e388ab2e43e97b3618363fbbe94ded","name":"Mi\u0161a Kruni\u0107","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/31aa4afb2464eca1f1ca0c7979628c87e54e7a6b53ebcb371749e9349d27c850?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/31aa4afb2464eca1f1ca0c7979628c87e54e7a6b53ebcb371749e9349d27c850?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/31aa4afb2464eca1f1ca0c7979628c87e54e7a6b53ebcb371749e9349d27c850?s=96&d=mm&r=g","caption":"Mi\u0161a Kruni\u0107"},"description":"Father of 2, Husband of 1, CEO of 3 :-)","sameAs":["http:\/\/www.price2spy.com"],"url":"https:\/\/www.price2spy.com\/blog\/author\/misha\/"}]}},"_links":{"self":[{"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/posts\/7300","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/comments?post=7300"}],"version-history":[{"count":3,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/posts\/7300\/revisions"}],"predecessor-version":[{"id":7416,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/posts\/7300\/revisions\/7416"}],"wp:attachment":[{"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/media?parent=7300"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/categories?post=7300"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.price2spy.com\/blog\/wp-json\/wp\/v2\/tags?post=7300"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}