You’ve just finished binge-watching Lupin on Netflix. What next? You browse “related” shows. Did you know that something as seemingly simple as this has a robust Machine Learning algorithm running behind the scenes?
Netflix has been obsessed with Machine Learning models for nearly two decades. The “Netflix Prize” was announced in 2006. The company offered a million dollars to anyone who could boost its existing algorithm’s recommendation accuracy by a mere 10%.
In 2009, Netflix declared a winner in this contest but junked the winning algorithm. Why?
Although the algorithm had superior accuracy, it was too complex. The engineering costs were very high, a trade-off that wasn’t worth the accuracy improvement. So, they ended up using a less expensive and simpler algorithm that delivered an 8.43% increase in accuracy.
Today, AI techniques are in demand in most industries. According to a McKinsey survey, 50% of organizations reported that they had adopted AI in at least one of their business functions.
However, unlike Netflix, many organizations fall prey to the accuracy fallacy. They focus too much on model accuracy and waste precious resources trying to over-optimize their algorithms. Just having an accurate model doesn’t guarantee business impact.
For your model to be successful, it must help deliver outcomes. Here are four ways to achieve that:
Contrast Model Accuracies With Human Performance
Numbers acquire meaning only when you place them in context. To contextualize model accuracy, benchmark it against the human error rate for the same activity in your organization. For instance, if human accuracy for a job is 80% and the model can deliver a 77% accuracy, it is just 3 percentage points off the mark.
Augment Model Performance With Human Intelligence
Rather than viewing Machine Learning solution models as replacements for humans, treat them as collaborators. Most algorithms will benefit by designing humans into the loop. In business scenarios that have insufficient data or high situational variability, humans can step in to stabilize and improve model performance. This is called augmented intelligence, and it consistently delivers better returns.
Assess How Much the Model Can Improve Over Time
Often, users don’t realize that model accuracies can improve over time. Machines learn and get better when they see more data. They improve with user feedback and with advances in research. For example, AlphaFold is the AI system from DeepMind to predict protein structures. It improved in accuracy from 68.5% in 2018 to 92.4% in 2020, thereby solving the challenge of protein folding.
Quantify the Outcome in Terms of Business Impact
Ask yourself what the real purpose of an ML model is. Can it help you increase revenues or slash costs? Translate the accuracy improvements into quantifiable metrics such as customers acquired or dollars saved. This will help you look beyond accuracy, talk business impact, and demonstrate value from machine learning.
In summary, model accuracy is not everything. At times, a model with a low accuracy may be your best bet to achieve your intended outcomes.
This is an excerpt from the article that originally appeared in Forbes. Read the full article that expands on these four points with industry examples and experiences shared by industry leaders.
