Crowd-sourcing with uncertain quality - an auction approach

Papakonstantinou, A. and Bogetoft, P. (2013): Crowd-sourcing with uncertain quality - an auction approach.

Abstract

This article addresses two important issues in crowd-sourcing: ex ante uncertainty about the quality and cost of different workers and strategic behaviour. We present a novel multi-dimensional auction that incentivises the workers to make partial enquiry into the task and to honestly report quality-cost estimates based on which the crowd-sourcer can choose the worker that offers the best value for money. The mechanism extends second score auction design to settings where the quality is uncertain and it provides incentives to both collect information and deliver desired qualities.

• Item Type: MPRA Paper
• Language: English
• Keywords: crowd-sourcing; Multi-dimensional auctions; Yardstick competition; Score functions; Strictly proper scoring rules
• Subjects: D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D86 - Economics of Contract: Theory; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D84 - Expectations ; Speculations; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D81 - Criteria for Decision-Making under Risk and Uncertainty; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D82 - Asymmetric and Private Information ; Mechanism Design

References

[1] D. R. Beil and L. Wein. An inverse-optimization-based auction mechanism to support a multiattribute rfq process. Management Science, pages 1529{1545, 2003.

[2] M. Bichler. An experimental analysis of multi-attribute auctions. Decision Support Systems, 29:249{268, 2000.

[3] M. Bichler and J. Kalagnanam. Confgurable offers and winner determination in multi-attribute auctions. European Journal of Operational Research, 160:380{394, 2003.

[4] P. Bogetoft and K. Nielsen. Dea based auctions. European Journal of Operational Research, 184:685{700, 2008.

[5] F. Branco. The design of multidimensional auctions. RAND Journal of Economics, 28(1):63{81, 1007.

[6] A. Charnes, W. W. Cooper, and E. Rhodes. Short communication: Measuring the efficiency of decision making units. European Journal of Operational Research, 3:339,1979.

[7] A. Charnes, W. W. Cooper, and E. Rhodes. Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6):429{444, 1987.

[8] Yeon-Koo Che. Design competition through multidimensional auctions. RAND Journal of Economics, 24(4):668{680, 1993.

[9] L. B. Chilton, J. J. Horton, R. C. Miller, and S. Azenkot. Task search in a human computation market. In Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP '10, pages 1{9, 2010.

[10] E. Clarke. Multipart pricing of public goods. Public Choice, 11(1):17{33, 1971.

[11] M. H. DeGroot and M. J. Schervish. Probability and Statistics. Addison Wesley, 2002.

[12] A. Doan, R. Ramakrishnan, and A. Y. Halevy. Crowdsourcing systems on the world-wide web. Communications of the ACM, 2011.

[13] E. Estelles-Arolas and F. Gonzalez-Ladron-de Guevara. Towards an integrated crowd-sourcing definition. Journal of Information Science, 38(2):189{200, 2012.

[14] D. Friedman. Effective scoring rules for probabilistic forecasts. Management Science,29(4):447{454, 1983.

[15] P. C. Gregory. Bayesian Logical Data Analysis for the Physical Sciences: A Comparative Approach with Mathematica Support. Cambridge University Press, 2005.

[16] T. Groves. Incentives in teams. Econometrica, 41(4):617{631, 1973.

[17] M. He, N. R. Jennings, and H.-F. Leung. On agent-mediated electronic commerce. IEEE Transactions on knowledge and data engineering, 15(4):985{1002, 2003.

[18] A. D. Hendrickson and R. J. Buehler. Proper scores for probability forecasters. The Annals of Mathematical Statistics, 42(6):1916{1921, 1971.

[19] A. Josang, R. Ismail, and C. Boyd. A survey of trust and reputation systems for online service provision. Decision Support Systems, 43(2):618{644, 2007.

[20] R. Jurca and B. Faltings. Reputation-based service level agreements for web services. In Service Oriented Computing, volume 3826 of Lecture Notes in Computer Science, pages 396{409. Springer Berlin / Heidelberg, 2005.

[21] R. Jurca and B. Faltings. Minimum payments that reward honest reputation feedback. In Proceedings of the ACM Conference on Electronic Commerce, pages 190{199, Michigan, USA, 2006.

[22] R. Jurca and B. Faltings. Collusion resistant, incentive compatible feedback payments. In Proceedings of the ACM Conference on Electronic Commerce, pages 200{209, San Diego, California, USA, 2007.

[23] A. Kittur, E. H. Chi, and S. Bongwon. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 453{456, 2008.

[24] A. Kittur, B. Smus, S. Khamkar, and R. E. Kraut. Crowdforge: crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology, UIST '11, pages 43{52, 2011.

[25] V. Krishna. Auction Theory. Academic Press, 2002.

[26] A. Mas-Collel, M. D. Whinston, and J. R. Green. Mircoeconomic Theory. Oxford University Press, 1995.

[27] N. Miller, P. Resnick, and R. Zeckhauser. Eliciting honest feedback: The peer prediction method. Management Science, 51(9):1359{1373, 2005.

[28] N. H. Miller, J. W. Pratt, R. J. Zeckhauser, and S. Johnson. Mechanism design with multidimensional, continuous types and interdependent valuations. Journal of Economic Theory, 136:476{496, 2007.

[29] A. Papakonstantinou. Mechanism Design for Eliciting Costly Observations in Next Generation Citizen Sensor Networks. PhD thesis, University of Southmapton, School of Electronic and Computer Science, 2010.

[30] A. Papakonstantinou, A. Rogers, E. H. Gerding, and N. R. Jennings. Mechanism design for the truthful elicitation of costly probabilistic estimates in distributed information systems. Artifficial Intelligence, 175(2):648{672, 2011.

[31] D. Parkes and J. Kalagnanam. Models for iterative multiattribute procurement auctions. Management Science, 51(3):435{451, 2005.

[32] Sarvapali D. Ramchurn, D. Huynh, and N. R. Jennings. Trust in multi-agent systems. The Knowledge Engineering Review, 19:1{25, 2004.

[33] L. J. Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783{801, 1977.

[34] R. Selten. Axiomatic characterization of the quadratic scoring rule. Experimental Economics, 1(1):43{61, 1998.

[35] W. Vickrey. Counterspeculation, auctions and competitive sealed tenders. The Journal of Finance, 16(1):8{37, 1961.

[36] J. Witkowski and D. C. Parkes. Peer prediction without a common prior. In Proceedings of the 13th ACM Conference on Electronic Commerce, EC '12, pages 964{981, 2012.

[37] J. Yang, L. A. Adamic, and M. S. Ackerman. Crowdsourcing and knowledge sharing: strategic user behavior on taskcn. In Proceedings of the 9th ACM conference on Electronic commerce, EC '08, pages 246{255. ACM, 2008.