How can a system use the data you provide to predict what you want to watch next on Netflix? Which purchase are you most likely to make on Amazon? Is there a way to cross-sell or up-sell to you?
Algorithms running recommender systems are one of the most powerful artificial intelligence applications running today. Data scientists need to choose the best one according to a business's limitations and requirements among a variety of recommendation algorithms. Before we look at the algorithms powering these systems, what is the underlining mathematical problem they have to solve?
What Will Your Next Move Be?
One of the most common types of algorithms for recommender systems is collaborative filtering (CF). These algorithms can be split into two senses, a narrow and a broader, more general sense. No matter the scope of these algorithms, they have to answer the following two steps:
Find out how many users/items in the database are similar to the given user/item.
Assess other users/items to predict what grade you would give the user of this product, given the total weight of the users/items that are more similar to this one.
In the narrow sense, the methodology is making automatic predictions (filtering) about a user's interests by collecting preferences from many users, thus collaborating. It is all about estimating how significant the differences in taste between people are. They try to narrow down and understand your preferences by using the information of many. It is in this regard, it differs from the more straightforward approach of giving an average (non-specific) score for each interest item, for example, based on its number of votes. It is, therefore, user-based collaborative filtering. If you are interested in a specific application, check out the Nearest Neighbor algorithm.
Within the more common sense, collaborative filtering is the method of sifting for data or designs utilizing methods including collaboration among different operators, perspectives, information sources, etc. Applications of collaborative filtering regularly include exceptionally expansive and complex data sets. These strategies are connected to numerous data types. Form sensing and monitoring data to financial data, algorithms in the realm of a more general sense have analyzed these vast amounts of data. In the broader sense, item-based collaborative filtering. Another method based on this principle is matrix factorization or matrix decomposition for recommendations.
Is Content King?
Content-based filtering approaches (item-based) utilize a series of discrete, pre-tagged characteristics of an item to recommend additional items with similar properties. The advantage of this is that these systems don't require vast amounts of user data. However, these applications are limited in scope.
With user-based collaborative filtering, it is the other way around. These systems need a lot of data to start making accurate recommendations. Not having enough relevant data available is a challenge most smaller businesses face. To overcome the challenge, innovative companies can turn to clustering, which is merging the two approaches. And, thus creating a hybrid recommendation system.
Things Recommendation Systems Need To Consider
There are other factors apart from accuracy when it comes to having successful recommender algorithms. I have picked some of the psychological ones I found quite interesting (for more, check out the resources below):
Recommendations should entail various products, songs, or movies that are not all directed by the same person, sung/composed by the same singer/composer, or produced from the same company. You get the idea: users prefer a more diverse display of recommendations.
Recommender persistence can be more helpful for the user experience of the system. Therefore, it can sometimes be more effective to show recommendations already shown, or let users re-rate items rather than showing them more products.
Trust - a recommender system is of little value for a user if they do not trust the system. A recommender system can build trust by explaining how it generates recommendations and recommends an item.
Serendipity is a measure of "how surprising the recommendations are." For instance, a recommender system that recommends milk to a customer in a grocery store might be entirely accurate. Still, it is not a good recommendation because it is an apparent item for the customer to buy.
This short summary should give a good overview of why we receive a similar compilation of film proposals now and then. When I think about it that way, I wish I had a more comprehensive selection. Aren't these systems restricting? What do you think, will there be something outside the mainstream in the future?
For this blog post, I have tried to understand the following resources:
Recommendation System Algorithms by Daniil Korbut
Hope you enjoyed the ride, and as always, stay curious!