Crowd-sourcing with uncertain quality - an auction approach

Papakonstantinou, A. and Bogetoft, P. (2013): Crowd-sourcing with uncertain quality - an auction approach.

Abstract

This article addresses two important issues in crowd-sourcing: ex ante uncertainty about the quality and cost of different workers and strategic behaviour. We present a novel multi-dimensional auction that incentivises the workers to make partial enquiry into the task and to honestly report quality-cost estimates based on which the crowd-sourcer can choose the worker that offers the best value for money. The mechanism extends second score auction design to settings where the quality is uncertain and it provides incentives to both collect information and deliver desired qualities.

• Item Type: MPRA Paper
• Language: English
• Keywords: crowd-sourcing; Multi-dimensional auctions; Yardstick competition; Score functions; Strictly proper scoring rules
• Subjects: D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D86 - Economics of Contract: Theory; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D84 - Expectations ; Speculations; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D81 - Criteria for Decision-Making under Risk and Uncertainty; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D82 - Asymmetric and Private Information ; Mechanism Design

References

[1] N. Archak. Money, glory and cheap talk: analyzing strategic behavior of contestants in simultaneous crowdsourcing contests on topcoder.com. In Proceedings of the 19th international conference on World wide web, WWW '10, pages 21{30}. ACM, 2010.

[2] D. R. Beil and L.Wein. An inverse-optimization-based auction mechanism to support a multiattribute rfq process. Management Science, pages 1529{1545}, 2003.

[3] M. Bichler. An experimental analysis of multi-attribute auctions. Decision Support Systems, 29:249{268}, 2000.

[4] M. Bichler and J. Kalagnanam. Congurable oers and winner determination in multi-attribute auctions. European Journal of Operational Research, 160:380{394},2003.

[5] P. Bogetoft and K. Nielsen. Dea based auctions. European Journal of Operational Research, 184:685{700}, 2008.

[6] F. Branco. The design of multidimensional auctions. RAND Journal of Economics, 28(1):63{81}, 1007.

[7] A. Charnes, W. W. Cooper, and E. Rhodes. Short communication: Measuring the effiency of decision making units. European Journal of Operational Research, 3:339, 1979.

[8] A. Charnes, W. W. Cooper, and E. Rhodes. Measuring the eciency of decision making units. European Journal of Operational Research, 2(6):429{444}, 1987.

[9] Yeon-Koo Che. Design competition through multidimensional auctions. RAND Journal of Economics, 24(4):668{680}, 1993.

[10] L. B. Chilton, J. J. Horton, R. C. Miller, and S. Azenkot. Task search in a human computation market. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '10, pages 1{9}, 2010.

[11] M. H. DeGroot and M. J. Schervish. Probability and Statistics. Addison Wesley, 2002.

[12] A. Doan, R. Ramakrishnan, and A. Y. Halevy. Crowdsourcing systems on the world-wide web. Communications of the ACM, 2011.

[13] E. Estelles-Arolas and F. Gonzalez-Ladron-de Guevara. Towards an integrated crowdsourcing denition. Journal of Information Science, 38(2):189{200}, 2012.

[14] D. Friedman. Eective scoring rules for probabilistic forecasts. Management Science, 29(4):447{454}, 1983.

[15] P. C. Gregory. Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica Support. Cambridge University Press, 2005.

[16] M. He, N. R. Jennings, and H.-F. Leung. On agent-mediated electronic commerce. IEEE Transactions on knowledge and data engineering, 15(4):985{1002}, 2003.

[17] A. D. Hendrickson and R. J. Buehler. Proper scores for probability forecasters. The Annals of Mathematical Statistics, 42(6):1916{1921}, 1971.

[18] A. Jsang, R. Ismail, and C. Boyd. A survey of trust and reputation systems for online service provision. Decision Support Systems, 43(2):618{644}, 2007.

[19] R. Jurca and B. Faltings. Reputation-based service level agreements for web services. In Service Oriented Computing, volume 3826 of Lecture Notes in Computer Science, pages 396{409}. Springer Berlin / Heidelberg, 2005.

[20] R. Jurca and B. Faltings. Minimum payments that reward honest reputation feedback. In Proceedings of the ACM Conference on Electronic Commerce, pages 190{199}, Ann Arbor, Michigan, USA, 2006.

[21] R. Jurca and B. Faltings. Collusion resistant, incentive compatible feedback payments. In Proceedings of the ACM Conference on Electronic Commerce, pages 200{209}, San Diego, California, USA, 2007.

[22] V. Krishna. Auction Theory. Academic Press, 2002.

[23] A. Mas-Collel, M. D. Whinston, and J. R. Green. Mircoeconomic Theory. Oxford University Press, 1995.

[24] N. Miller, P. Resnick, and R. Zeckhauser. Eliciting honest feedback: The peer prediction method. Management Science, 51(9):1359{1373}, 2005.

[25] N. H. Miller, J. W. Pratt, R. J. Zeckhauser, and S. Johnson. Mechanism design with multidimensional, continuous types and interdependent valuations. Journal of Economic Theory, 136:476{496}, 2007.

[26] A. Papakonstantinou. Mechanism Design for Eliciting Costly Observations in Next Generation Citizen Sensor Networks. PhD thesis, University of Southmapton, School of Electronic and Computer Science, 2010.

[27] A. Papakonstantinou, A. Rogers, E. H. Gerding, and N. R. Jennings. Mechanism design for the truthful elicitation of costly probabilistic estimates in distributed information systems. Articial Intelligence, 175(2):648{672}, 2011.

[28] D. Parkes and J. Kalagnanam. Models for iterative multiattribute procurement auctions. Management Science, 51(3):435{451}, 2005.

[29] Sarvapali D. Ramchurn, D. Huynh, and N. R. Jennings. Trust in multi-agent systems. The Knowledge Engineering Review, 19:1{25}, 2004.

[30] L. J. Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783{801, 1977.

[31] R. Selten. Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 1(1):43{61}, 1998. [32] A. Sorokin and D. Forsyth. Utility data annotation with amazon mechanical turk. CVPR '08, 2008.

[33] J. Witkowski and D. C. Parkes. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce, EC '12, pages 964{981},2012.

[34] J. Yang, L. A. Adamic, and M. S. Ackerman. Crowdsourcing and knowledge sharing: strategic user behavior on taskcn. In Proceedings of the 9th ACM conference on Electronic commerce, EC '08, pages 246{255}. ACM, 2008.