Decision Making Using Rating Systems: When Scale Meets Binary

Bargagliotti, Anna E. and Li, Lingfang (Ivy) (2009): Decision Making Using Rating Systems: When Scale Meets Binary.

Abstract

Rating systems measuring quality of products and services (i.e., the state of the world) are widely used to solve the asymmetric information problem in markets. Decision makers typically make binary decisions such as buy/hold/sell based on aggregated individuals' opinions presented in the form of ratings. Problems arise, however, when different rating metrics and aggregation procedures translate the same underlying popular opinion to different conclusions about the true state of the world. This paper investigates the inconsistency problem by examining the mathematical structure of the metrics and their relationship to the aggregation rules. It is shown that at the individual level, the only scale metric (1,. . . ,N) that reports people's opinion equivalently in the a binary metric (-1, 0, 1) is one where N is odd and N-1 is not divisible by 4. At aggregation level, however, the inconsistencies persist regardless of which scale metric is used. In addition, this paper provides simple tools to determine whether the binary and scale rating systems report the same information at individual level, as well as when the systems di®er at the aggregation level.

• Item Type: MPRA Paper
• Language: English
• Keywords: rating, ranking, preference, asymmetric information
• Subjects: D - Microeconomics > D7 - Analysis of Collective Decision-Making > D70 - General; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D82 - Asymmetric and Private Information ; Mechanism Design

References

[1]Alwin, F. D. and Krosnick, A. J. (1985). The Measurement of Values in Surveys: A Comparison of Ratings and Rankings, The Public Opinion Quarterly, 49 (4), 535-552.

[2] Arrow, K. J. (1963). Social Choice and Individual Values, John Wiley Sons, Inc., New York, London, Sydney.

[3] Cleveland, J. N. and Murphy, K. R., (1992). Analyzing performance appraisal as goal-directed behavior. In G. Ferris K. Rowland (Eds.), Research in personnel and human resources man- agement (Vol. 10, pp. 121185). Greenwich, CT: JAI Press.

[4] VP Crawford and J Sobel, (1982). Strategic Information Transmission. Econometrica, 50, 1431-1451.

[5] C. Dellarocas, F. Dini, and G. Spagnolo. (2006) Designing reputation (feedback) mechanisms. In Handbook of Procurement, Cambridge University Press, 2006.

[6] Dellarocas, C and Wood, C. A. The sound of silence in online feedback: Estimating trading risks in the presence of reporting bias. Management Science, Forthcoming.

[7] Droba, D. D. (1931). Methods Used for Measuring Public Opinion, The American Journal of Sociology, 73 (3), 410-423.

[8] Greenleaf,E A. , B. Bickart, and E. A. Yorkston, (1999), How Response Styles Weaken Corre- lations from Rating Scale Surveys, Working paper, Stern School of Business, New York

[9] Klein,T. J., C. Lambertz, G. Spagnolo, and K. O. stahl. (2006) Last minute feedback. CEPR Discussion Paper No. 5693. (2006)

[10] Landy, F. J. and Farr, J. L., (1980), Performance Rating, Psychological Bulletin, 87 (1), 72-107.

[11] Lehmann, D. and Hulbert, J., (1972). Are Three-Point Scales Always Good Enough?, Journal of Marketing Research 9, (4), 444-446.

[12] Li, L. I., "Reputation, Trust, and Rebates: How Online Auction Markets Can Improve Their Feedback Systems," Journal of Economics and Management Strategy, (forthcoming) .

[13] Li, L. I. and Saari, D. G. (2008). Sens theorem: geometric proof, new interpretations, Social Choice and Welfare, 31 (2008), 393-413.

[14] McCarty, J. and Shrum, L.J., (2000). The Measurement of Personal Values in Survey Research: A Test of Alternative Rating Procedures, The Public Opinion Quarterly, 64, (3), 271-298.

[15] McDaniel, C. and Gates, R., (2004). Marketing Research, Wiley; 6 edition

[16] Morgan, J and Stocken, P. (2008). Information Aggregation in Polls. American Economic Review, 98, (3), 864-896.

[17] Murphy, K. R. , J. N. Cleveland, A. L. Skattebo, and T. B. Kinney, (2004), Raters Who Pursue Different Goals Give Different Ratings, Journal of Applied Psychology, 89 (1), 158-164.

[18] Saari, D.G. (1999). Explaining all three alternative voting outcomes, Journal of Economic Theory, 87, 313-355 [19] Saari, D.G. (2001a) Chaotic elections! A mathematician looks at voting Providence, American Mathematical Society.

[20] Saari, D. G. (2001b). Decisions and Elections; Explaining the Unexpected, Cambridge Univer- sity Press, New York.

[21] Schwarz, N. , B. Knaper, H. Hippler, E. Noelle-Neumann, and L. Clark, (1991), Rating Scales: Numeric Values May Change the Meaning of Scale Labels, The Public Opinion Quarterly, 55 (4), 570-582.

[22] Sen, A. K. (1970a). Collective Choice and Social Welfare, Holden-Day, San Francisco.

[23] Sen, A. K. (1970b). The Impossibility of a Paretian Liberal, The journal of Political Economy, 78(1), 152-57.

[24] Sen, A. K. (1986) Social Choice Theory; in Handbook of Mathematical Economics, Vol. III; Ed. K.J. Arrow and M. Intriligator, Amsterdam: North-Holland.

[25] Sobel, J.. (2006) Information Aggregation and Group Decisions, Working Paper, University of California, San Diego.

[26] Stevens, S. S. (1946). On the Theory of Scales of Measurement, Science, 103 (2684), 677-680