What is a Fair Value of Your Recommendation List?

Bobrikov, Vladimir and Nenova, Elena and Ignatov, Dmitry I. (2016): What is a Fair Value of Your Recommendation List? Published in: Proceedings of the Third Workshop on Experimental Economics and Machine Learning , Vol. 1627, No. urn:nbn:de:0074-1627-1 (25 July 2016): pp. 1-12.

Abstract

We propose a new quality metric for recommender systems. The main feature of our approach is the fact, that we take into account the set of requirements, which are important for business application of a recommender. Thus, we construct a general criterion, named “audience satisfaction”, which thoroughly describe the result of interaction between users and recommendation service. During the criterion construction we had to deal with a number of common recommenders’ problems: a) Most of users rate only a random part of the objects they consume and a part of the objects that were recommended to them; b) Attention of users is distributed very unevenly over the list of recommendations and it requires a special behavioral model; c) The value of the user’s rate measures the level of his/her satisfaction, hence these values should be naturally incorporated in the criterion intrinsically; d) Different elements may often dramatically differ from each other by popularity (long tail – short head problem) and this effect prevents accurate measuring of user’s satisfaction. The final metric takes into account all these issues, leaving opportunity to adjust the metric performance based on proper behavioral models and parameters of short head problem treatment.

• Item Type: MPRA Paper
• Language: English
• Keywords: recommender systems, quality metric, explicit feedback, movie recommendations, AUC, cold start, recommendations for novices
• Subjects: C - Mathematical and Quantitative Methods > C0 - General; C - Mathematical and Quantitative Methods > C0 - General > C02 - Mathematical Methods; C - Mathematical and Quantitative Methods > C1 - Econometric and Statistical Methods and Methodology: General > C10 - General; D - Microeconomics > D4 - Market Structure, Pricing, and Design > D46 - Value Theory; Y - Miscellaneous Categories > Y1 - Data: Tables and Charts > Y10 - Data: Tables and Charts

References

Gunawardana, A., Shani, G.: Evaluating recommender systems. In: Recommender Systems Handbook. (2015) 265–308

Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: Proceedings of the 2010 ACM Conference on Recommender Systems, RecSys 2010, Barcelona, Spain, September 26-30, 2010. (2010) 39–46

Knijnenburg, B.P., Willemsen, M.C.: Evaluating recommender systems with user experiments. In: Recommender Systems Handbook. (2015) 309–352

Adamopoulos, P., Tuzhilin, A.: On unexpectedness in recommender systems: Or how to better expect the unexpected. ACM Trans. Intell. Syst. Technol. 5 (4) (December 2014) 54:1–54:32

Schein, A.I., Popescul, A., Ungar, L.H., Pennock, D.M.: CROC: A new evaluation criterion for recommender systems. Electronic Commerce Research 5 (1) (2005) 51–74

Lu, Q., Chen, T., Zhang, W., Yang, D., Yu, Y.: Serendipitous personalized ranking for top-n recommendation. In: 2012 IEEE/WIC/ACM International Conferences on Web Intelligence, WI 2012, Macau, China, December 4-7, 2012. (2012) 258–265

Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. Trans. Inform. Syst. 27 (December 2008)

Davis, J., Goadrich, M.: The relationship between precision-recall and roc curves. In: ICML ’06 Proceedings of the 23rd international conference on Machine learning. (2006) 233–240 ISBN:1-59593-383-2.

Pilászy, I., Tikk, D.: Recommending new movies: even a few ratings are more valuable than metadata. In: Proceedings of the 2009 ACM Conference on Recommender Systems, RecSys 2009, New York, NY, USA, October 23-25, 2009. (2009) 93–100