Simple Belief Elicitation: an experimental evaluation

Schlag, Karl and Tremewan, James (2020): Simple Belief Elicitation: an experimental evaluation.

Abstract

We present a method for eliciting beliefs about probabilities when multiple realisations of an outcome are available, the "frequency'' method. The method is applicable for any reasonable utility function. Unlike existing techniques that account for deviations from risk-neutrality, this method is highly transparent to subjects and easy to implement. Rather than identifying point beliefs these methods identify bounds on beliefs, thus trading off precision for generality and simplicity. An experimental comparison of this method and a popular alternative, the Karni method, shows that subjects indeed find the frequency method easier to understand. Significantly, we show that confusion due to the complexity of the Karni method leads to less cognitively able subjects erroneously stating a belief of 50%, a bias not present in the frequency method.

• Item Type: MPRA Paper
• Language: English
• Keywords: Belief elicitation; experiment
• Subjects: C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C91 - Laboratory, Individual Behavior

References

Bhatt, M. and C. F. Camerer (2005). Self-referential thinking and equilibrium as states of mind in games: fmri evidence.

Blanco, D. Engelmann, A. K. Koch, and H.-T. Norman (2010). Belief elicitation in experiments: is there a hedging problem?

Burfurd, I. and T. Wilkening (2018). Experimental guidance for eliciting beliefs with the Stochastic Becker DeGrootMarschak mechanism.

Burfurd, I. and T. Wilkening (2020). Cognitive heterogeneity and complexbelief elicitation. mimeo.

Cosmides, L. and J. Tooby (1996). Are humans good intuitive statisticians after all? rethinking some conclusions from the literature on judgement under uncertainty

Costa-Gomes, M. A. and G. Weizsacker (2008). Stated beliefs and play in normal-form games

Dal Bo, E., P. Dal Bo, and E. Eyster (2017). The demand for bad policy when voters underappreciate equilibrium effects.

Gigerenzer, G. and U. Hoffrage (1995). How to improve bayesian reasoning without instruction: Frequency formats.

Greiner, B. (2015). Subject pool recruitment procedures: organizing experiments with orsee.

Hurley, T. M. and J. F. Shogren (2005). An experimental comparison of induced and elicited beliefs.

Kahneman, D. and A. Tversky (1983, October). Extensional versus intuitivereasoning: The conjunction fallacy in probability judgement.

Karni, E. (2009). A mechanism for eliciting probabilities.

Le Coq, C., J. Tremewan, and A. K. Wagner (2015). On the effects of group identity in strategic environments.

Lipkus, I. M., G. Samsa, and B. K. Rimer (2001). General performance on a numeracy scale among highly educated samples.

Manski, C. F. (2004, September). Measuring expectations.

Offerman, T., J. Sonnemans, G. Van de Kuilen, and P. P. Wakker (2009). A truth serum for non-bayesians: Correcting proper scoring rules for risk attitudes.

Powell, O. (2019). jtree - a javascript toolbox for running economics experiments. https://opowell.github.io/jtree.

Schapira, M. M., A. B. Nattinger, and C. A. McHorney (2001). Frequency or probability? a qualitative study of risk communication formats used in health care.

Schlag, K. H. (2015). Who gives direction to statistical testing? best practice meets mathematically correct tests. Best Practice Meets Mathematically Correct Tests.

Schlag, K. H. and J. Tremewan (2014). Simple belief elicitation.

Schlag, K. H., J. Tremewan, and J. J. Van der Weele (2015). A penny for your thoughts: A survey of methods for eliciting beliefs.

Schotter, A. and I. Trevino (2014). Belief elicitation in the laboratory.

Schwarz, L. M., S. Woloshin, W. C. Black, and H. G. Welch (1997). The role of numeracy in understanding the benefit of screening mammography.

Tversky, A. and D. Kahneman (1974). Judgment under uncertainty: Heuristics and biases.

Wilcox, N. T. and N. Feltovich (2000). Thinking like a game theorist: Comment.