{"id":7297,"date":"2020-06-19T11:35:05","date_gmt":"2020-06-19T11:35:05","guid":{"rendered":"https:\/\/www.price2spy.com\/blog\/?p=7297"},"modified":"2020-07-28T10:50:42","modified_gmt":"2020-07-28T10:50:42","slug":"part-5-ml-training-implementation","status":"publish","type":"post","link":"https:\/\/www.price2spy.com\/blog\/part-5-ml-training-implementation\/","title":{"rendered":"(Part #5) ML training Implementation"},"content":{"rendered":"\n

Product matching in Price2Spy <\/a><\/p>\n\n\n\n

Previous topic: <\/strong> (Part #4) Preparing the ML training set<\/a> <\/p>\n\n\n\n

Next topic: <\/strong>(Part #6) Evaluating ML training results <\/a><\/p>\n\n\n\n

Our very first implementation step was a bit non-standard. While most of ML is done in Python, Price2Spy<\/a> is a Java shop. We respect other technologies, but our love and our choice go to Java. And finding reliable ML libraries in Java is no piece of cake (though I must say, as months go by, more and more Java enthusiasts are posting ML libraries in the public domain).<\/p>\n\n\n\n

Pre-processing<\/h3>\n\n\n\n

Next problem: sheer numbers. Product matching<\/a> has one nasty logical thing about it. A match is a pair \u2013 a product A is (or is not) a match with Product B. And what if Set A has 50K products, while Set B has 40K? <\/p>\n\n\n\n

That means 50K x 40K = 2G (2 000 000 000) combinations to be checked! And for each combination, we\u2019ll have to perform 30-40 computing operations (called \u2018features\u2019). Even with modern CPU power, that\u2019s just too much (we have excluded cloud solution very early since the cost would be just too high). <\/p>\n\n\n\n

A simple calculation said that our training process (just one single iteration) would take 3.5 years!<\/p>\n\n\n\n

\"ML<\/figure><\/div>\n\n\n\n

This is where we had to pull out the first ML trick \u2013 it\u2019s called \u2018blocking\u2019. What blocking practically does (by performing operations which are not that heavy on CPU) is that it eliminates matching pairs which are too improbable. We have made blocking in a configurable way, so we were able to put a threshold of what is considered improbable enough. <\/p>\n\n\n\n

Thanks to blocking, we have reduced 2.0G to 1.8M matching combinations to be checked. So, from 3.5 years, we get to 20 hours CPU time. What a relief! And not only have we reduced the sheer number of combinations, we have also ensured that our training set contains both good matches and wrong matches which are not that improbable \u2013 which means better learning examples.<\/p>\n\n\n\n

However, that\u2019s not all \u2013 we\u2019re living in the real world, and in reality, data is not 100% clean. For example, the same product might be listed twice on a particular website \u2013 under 2 different categories. This is very dangerous for ML, because if it sees it as 2 separate products \u2013 and only one of them will be matched, while both of them have the same characteristics. This means \u2018noise\u2019 which results in lower accuracy.<\/p>\n\n\n\n

The answer to this was another ML technique called \u2018deduplication\u2019 \u2013 thank God, that was much easier than blocking.<\/p>\n\n\n\n

ML feature set<\/h3>\n\n\n\n

The next thing to tackle was \u2013 which features to use. Please note that we consider this as the heart of our ML project, so we won\u2019t be able to reveal all \u2018secret ingredients\u2019. What we can say that the number of features is around 40 and that they can be categorized as:<\/p>\n\n\n\n