A.I. recommendation engines are everywhere, and you encounter them every day. They are machine learning models that predict your preference in the movies you watch, the songs you listen to, the products you buy, and even the people you date.
Netflix uses recommendation engines to predict what movies or television shows you may be interested in viewing. Netflix claims that 75% of what people watch on Netflix is from recommendations. Spotify uses recommendation engines to predict the songs you may enjoy, especially newly released tunes. Spotify claims that the rise in users from 75M to 100M, at a time when faced with huge competition, was due to their superior recommendation engine. Amazon uses recommendation engines to predict what you may be interested in buying, and attributes 35% of all their retail sales to their now-famous “people who bought this item also bought” recommendations. Even dating sites such as Tinder use recommendation engines to suggest potential mates.
There are five main types of filters used in recommendation engines that are the most widely employed. There are also hybrid approaches that utilize some or all of the five main filters.
Collaborative filtering assumes “birds of a feather flock together,” and makes predictions based on similarities in taste preferences. For example, if Bob liked Blade Runner, American Made, and Flatliners, and Sue liked Blade Runner and American Made, then Sue may also like Flatliners. This is one of top two most popular recommendation models, simply because it works so effectively. The proof of the effectiveness of this model is antidotal in everyday life. Let’s say you love drama movies, and your friend Jane loves action movies and your fiend Mary also loves drama movies. If Jane and Mary both recommend different movies for you to see, you are most likely to follow Mary’s recommendations you both enjoy drama movies.
Collaborative filtering works great, but sometimes there is just not enough data in similar behavioral patterns to create a recommendation. Or, there may be much richer correlations in similar content than collaboration. This is where content-based filtering comes into play. Content-based filtering assumes I will like similar things to the things I already like. For example, if I liked Star Wars, I may like Star Trek. Or if I like pizza, I may also like pasta. Content-based filtering is not usually as broad-stroked as the previous examples. It often assigns dozens to hundreds of features to an observation. This allows for a more accurate prediction. For example, while pizza and pasta are both Italian dishes, pizza can be eaten with your hands and pasta is eaten with a fork, as well as many other fine differences between pizza and pasta.
The image below visually shows the differences between collaborative filtering and content-based filtering.
Demographic-based filtering is another take on the “birds of a feather flock together” assumption. We naturally assume that a 17-year old boy will have different tastes than a 64-year-old woman; that a 64-year-old woman will have somewhat similar preferences to a 64-year-old man, but even more similar preferences to another 64-year-old woman. Demographic-based filtering allows the recommendation engine to suggest products that may not have been purchased by the same group, nor have similar qualities. For example, if 64-year-old Ben likes Acme Vitamins. And 64-year-old Mary likes Acme Vitamins, we can recommend Acme Vitamins to 64-year-old Sue, regardless of what else she purchased. The more granular the demographics, the more accurate the prediction usually is. This type of filtering has a dark side in racial profiling, and is indeed used by the NSA and FBI. However, the fact remains that demographic similarities typically yield accurate recommendations. And, there is wide swath of demographics, including age, gender, race, religion, political views, career, hobbies, sports, income bracket, and geographical location…just to name a few. Just think about all the people you know in one or more of the aforementioned demographics with whom you may share similarities in tastes.
Utility-based filtering is one of the oldest and simplest filtering. It is somewhat powerful, but in the broader perspective, not nearly as powerful as collaborative, content-based, and demographics-based filtering. A utility is based on usefulness to a consumer. The simplest example is the filtering of automobiles. A couple with a new baby may need a 4-door vehicle, and perhaps a minivan, if planning additional users to the family. The utility, or usefulness, of the product becomes the number one determinant for recommendation in utility-based filtering. Utility-based filtering is most often used in conjunction with other filtering methods. More on this in just a bit.
Knowledge-based filtering creates recommendations based on the knowledge of an individual’s preferences and needs. A bit of a hybrid filtering engine, this model must be aware of what the user likes as well as what the user needs. For example, if a hurricane is approaching the user’s geolocation, and the user has purchased canned beans in the past, a knowledge-based filtering engine may recommend the user stock up on canned foods, and specifically beans. As mentioned, knowledge-based filtering is a bit of a hybrid system, which produce superior results to any single method of filtering.
By far, the most power recommendation engines are ensembles. They are hybrids of the most effective recommendation filtering engines. They use a combination of collaboration, content-based, demographic-based, utility, knowledge, and even reinforcement learning (which is out of scope for this article; we will discuss Reinforcement Learning in another article soon). In 2009, Netflix offered $1,000,000 to anyone who could create a recommendation engine that is just 10% better than their current engine. Many came close to the 10% mark with single-focused filtering, like collaboration and content-based filtering. It was when several teams pooled their resources and combined their proprietary single-focus filters into an ensemble of filters, results in a score that surpassed 10%, and won the $1,000,000 Netflix prize.
The image below depicts an ensemble of collaboration and demographics-based filtering.
Netflix has succeeded in becoming the world dominant streaming movie provider. They owe their success to their recommendation engine. That’s right! It’s not their movies, their television shows, or even their original content. It’s that the recommendations they make are consistent with what you want to watch. This creates a stickiness, and contributes to brand loyalty. After all, why go to Hulu and have to retrain all of your preferences when Netflix already knows what you like?
Recommendation engines are a very powerful implementation of A.I. that can spell the difference between success and failure of a business. A good recommendation engine is what I deem one of the “unfair competitive advantages” of machine learning.
Do you want to utilize the benefits of an unfair competitive advantage? Contact us to learn how A.I. can help your business grow and dominate.
P.S. – Netflix has a standing offer of $10,000,000 to anyone that can create a next-generation recommendation system. If you do, however, I suggest shopping it around to others like Amazon and Spotify…they are all willing to pay top-dollar for an unfair competitive advantage.