Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared

Green, Kesten C.; Armstrong, J. Scott and Graefe, Andreas (2007): Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared. Forthcoming in: Foresight: The International Journal of Applied Forecasting No. Fall

Abstract

Traditional groups meetings are an inefficient and ineffective method for making forecasts and decisions. We compare two structured alternatives to traditional meetings: the Delphi technique and prediction markets. Delphi is relatively simple and cheap to implement and has been adopted for diverse applications in business and government since its origins in the 1950s. It can be used for nearly any forecasting, estimation, or decision making problem not barred by complexity or ignorance. While prediction markets were used more than a century ago, their popularity waned until more recent times. As a consequence there is less evidence on their validity. Prediction markets need many participants. They need clear outcomes in order to determine participants’ pay-offs. Even so, relating their knowledge to market prices is not intuitive to everyone and constructing contracts that will provide a useful forecast may not be possible for some problems. It is difficult to maintain confidentiality with markets and they are vulnerable to manipulation. Delphi is designed to reveal panelists’ knowledge and opinions via their forecasts and the reasoning they provide. This format allows testing of knowledge and learning by panelists as they refine their forecasts. Such a process does not happen explicitly in prediction markets and may not happen at all. The reasoning provided as an output of the Delphi process is likely to be reassuring to forecast users who are uncomfortable with the “black box” nature of prediction markets. We consider that, half a century after its original development, Delphi is greatly under-utilized.

• Item Type: MPRA Paper
• Institution: Monash University Business and Economic Forecasting Unit
• Language: English
• Keywords: accuracy; forecasting methods; groups; judgment; meetings; structure
• Subjects: C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology; Computer Programs > C88 - Other Computer Software; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D84 - Expectations; Speculations; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C42 - Survey Methods; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D82 - Asymmetric and Private Information; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C49 - Other; C - Mathematical and Quantitative Methods > C4 - Econometric and Statistical Methods: Special Topics > C44 - Statistical Decision Theory; Operations Research; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D81 - Criteria for Decision-Making under Risk and Uncertainty; D - Microeconomics > D8 - Information, Knowledge, and Uncertainty > D83 - Search; Learning; Information and Knowledge; Communication; Belief

References

Abramowicz, M. B. (2004). Information Markets, Administrative Decisionmaking, and Predictive Cost-Benefit Analysis. University of Chicago Law Review, 71, 933-1020. Armstrong, J. S. (2006). How to Make Better Forecasts and Decisions: Avoid Face-to-Face Meetings. Foresight, 5, 3-8. Chen, K.-Y. & Plott, C. R. (2002). Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem. Social Science Working Paper No.1131, California Institute of Technology, Pasadena. Gordon, T. & Pease, A. (2006). RT Delphi: An Efficient, “Round-Less” Almost Real Time Delphi Method. Technological Forecasting and Social Change, 73, 321-333. Green, K. C. & Armstrong, J. S. (2007). The Value of Expertise for Forecasting Decisions in Conflicts. Interfaces, 37, 287-299. Hoffmann, S., Fischbeck, P., Krupnick, A. & McWilliams, M. (2007). Elicitation from Large, Heterogeneous Expert Panels: Using Multiple Uncertainty Measures to Characterize Information Quality for Decision Analysis. Decision Analysis, 4 (2), 91-109. King, R. (2006). Workers, Place Your Bets, BusinessWeek, August 3, http://www.businessweek.com/technology/content/aug2006/tc20060803_012437.htm. Looney, R. E. (2004). DARPA's Policy Analysis Market for Intelligence: Outside the Box or Off the Wall? International Journal of Intelligence and Counterintelligence, 17, 405-419. Rhode, P. W. & Strumpf, K. S. (2004). Historical Presidential Betting Markets. The Journal of Economic Perspectives, 18, 127-141. Rowe, G. (2007). A Guide to Delphi. Foresight, 8, forthcoming. Rowe, G. & Wright, G. (2001). Expert opinions in Forecasting: The Role of the Delphi Technique. In: J. S. Armstrong (Ed.), Principles of Forecasting - A Handbook for Researchers and Practitioners. Boston, MA; Kluwer Academic Publishers, pp. 125-144. Wolfers, J. & Zitzewitz, E. (2006). Prediction Markets in Theory and Practice. NBER Working Paper 12083, http://bpp.wharton.upenn.edu/jwolfers/Papers/PredictionMarkets(Palgrave).pdf.