Cognitive Reflection Test: Whom, how, when

Brañas-Garza, Pablo and Kujal, Praveen and Lenkei, Balint (2015): Cognitive Reflection Test: Whom, how, when.

Abstract

We report the results of a meta-study of 118 Cognitive Reflection Test studies comprising of 44,558 participants across 21 countries. There is a negative correlation between being female and the overall,and individual, correct answers to CRT questions. Taking the test at the end of an experiment negatively impacts performance. Monetary incentives do not impact performance. Overall students perform better compared to non-student samples. Exposure to CRT over the years may impact outcomes, however, the effect is driven by online studies. We obtain mixed evidence on whether the sequence of questions matters. Finally, we find that computerized tests marginally improve results.

• Item Type: MPRA Paper
• Language: English
• Keywords: CRT, Experiments, Gender, Incentives, Glucose and Cognition.
• Subjects: Z - Other Special Topics > Z0 - General > Z00 - General

References

Aiken, L. (1986-1987). Sex differences in mathematical ability: A review of the literature. Educational Research Quarterly, 10: 25-35.

Alós-Ferrer, C., & Hügelschäfer, S. (2014). Faith in Intuition and Cognitive Reflection. University of Cologne Working Paper. Study 3.

Andersson, O., Tyran, J.R., Wengström, E., & Holm, H.J. (2013). Risk Aversion Relates to Cognitive Ability: Fact or Fiction? IFN Working Paper No. 964.

Awasthi, V., & Pratt, J. (1990). The Effects of Monetary Incentives on Effort and Decision Performance: The Role of Cognitive Characteristics. The Accounting Review, 65(4): 797-811.

Benbow, C.P., & Stanley, J.C. (1980). Sex Differences in Mathematical Ability: Fact or Artifact? Science, 210(4475): 1262–264.

Benbow, C. P., Lubinski, D., Shea, D. L., & Eftekhari-Sanjani, H. (2000). Sex differences in mathematical reasoning ability: Their status 20 years later. Psychological Science, 11: 474-480.

Ben-Ner, A., Kong, F., & Putterman, L. (2004). Share and Share Alike? Gender-pairing, Personality, and Cognitive Ability as Determinants of Giving. Journal of Economic Psychology, 25(5): 581-589.

Besedes, T., Deck, C., Sarangi, S., & Shor, M. (2012). Decision-making Strategies and Performance among Seniors. Journal of Economic Behavior and Organization, 81(2): 524-533.

Bonner, S.E., Sprinkle, G.B. (2002) The effects of monetary incentives on effort and task performance:theories, evidence, and a framework for research. Accounting, Organizations and Society, 27: 303–345.

Bosch-Rosa, C., Meissner, T., & Bosch-Domenech, A. (2015). Cognitive Bubbles. SFB 649 Discussion Paper 2015-006.

Buhrmester, M., Kwang, T., & Gosling, S.D. (2011). Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1): 3-5.

Brañas-Garza, P., García-Muñoz, T., & González, R.H. (2012). Cognitive effort in the Beauty Contest Game. Journal of Economic Behavior & Organization, 83(2): 254–260.

Camerer, C. F., & Hogarth, R.M. (1999). The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework. Journal of Risk and Uncertainty, 19(1): 7-42.

Campitelli, G., & Labollita, M. (2010). Correlations of cognitive reflection with judgments and choices.Judgment and Decision Making, 5(3): 182-191.

Carpenter, J., Graham, M., & Wolf, J. (2013). Cognitive ability and strategic sophistication. Games and Economic Behavior, 80(1): 115–130.

Chen, C.C., Chiu, I.M., Smith, J., & Yamada, T. (2013). Too smart to be selfish? Measures of cognitiveability, social preferences, and consistency. Journal of Economic Behavior & Organization, 90(0): 112–122.

Cheung, S. L., Hedegaard, M., & Palan, S. (2014). To See is to Believe. Common Expectations in Experimental Asset Markets. European Economic Review, 66: 84–96.

Cole, M.S., Bedeian, A.G., & Field, H.S. (2006). The Measurement Equivalence of Web-Based and Paper-and-Pencil Measures of Transformational Leadership: A Multinational Test. Organizational Research Methods, 9(3): 339-368.

Corgnet, B., Espín, A.M., & Hernán-González, R. (2015). The cognitive basis of social behavior: cognitive reflection overrides antisocial but not always prosocial motives. ESI Working Paper 15-04.

Corgnet, B., Espin, A., Hernan-Gonzalez, R., Kujal, P., & Rassenti, S. (2015). To trust, or not to trust: Cognitive reflection in trust games. Forthcoming in Journal of Behavioral & Experimental Economics.

Corgnet, B., Hernan-Gonzalez, R., Kujal, P., & Porter, D. (2014). The Effect of Earned Versus House Money on Price Bubble Formation in Experimental Asset Markets, Review of Finance, 1-34. [doi:10.1093/rof/rfu031]

Cueva-Herrero, C., Iturbe-Ormaetxe, I., Mata-Prez, E., Ponti, G., Yu, H., & Zhukova, V. (2015). Cognitive (Ir)reflection: New Experimental Evidence. Forthcoming in Journal of Behavioral and Experimental Economics.

Diamond, D.W., & Dybvig, P.H. (1983). Bank Runs, Deposit Insurance, and Liquidity. Journal of Political Economy, 91(3): 401-419.

Epstein, S. (1994). Integration of the cognitive and the psychodynamic unconscious. American Psychologist, 49: 709–724.

Exadaktylos, F., Espín, A.M., & Branas-Garza, P. (2013). Experimental subjects are not different. Scientific Reports, 3(1213): 1-6.

Falk, A., & Heckman, J. (2009). Lab experiments are a major source of knowledge in the social sciences. Science, 326: 535–538.

Falk, A., Meier, S., & Zehnder, C. (2013). Do lab experiments misrepresent social preferences? The case of self-selected student samples. Journal of the European Economic Association, 11(4): 839–852.

Frederick, S. (2005). Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19(4): 25-42.

Gailliot, M.T., & Baumeister, R.F. (2007). The Physiology of Willpower: Linking Blood Glucose to Self-Control. Personality and Social Psychology Review, 11(4): 303-327.

George, C.E., Lankford, J.S., & Wilson, S.E. (1992). The effects of computerised versus paper-and- pencil administration on measures of negative affect. Computers in Human Behavior, 8(2-3): 203-209.

Gneezy, U., & Rustichini, A. (2000a). Pay enough or don’t pay at all. Quarterly Journal of Economics, 115(3): 791–810.

Goodman, J. K., Cryder, C.E., & Cheema, A.A. (2013). Data Collection in a Flat World: Strengths and Weaknesses of Mechanical Turk Samples. Journal of Behavioral Decision Making, 26: 213-224.

Holt, C.A, Porzio, M., & Song, M.Y. (2015). Price Bubbles, Expectations, and Gender in Asset Markets: An Experiment. University of Virginia working paper.

Hoppe, E. I., & Kusterer, D.J. (2011). Behavioral biases and cognitive reflection. Economics Letters, 110: 97–100.

Jenkins, G. D., Mitra, A., Gupta, N., & Shaw, J.D. (1998). Are financial incentives related to performance? A meta-analytic review of empirical research. Journal of Applied Psychology, 83(5): 777-787.

Kahneman, D., & Frederick, S. (2002). Representativeness revisited: Attribute substitution in intuitive judgment. In T. Gilovich, D. Griffin and D. Kahneman (Eds.), Heuristics and biases: The psychology of intuitive judgment, 49-81, New York: Cambridge University Press.

King, W.C., & Miles, E.W. (1995). A quasi-experimental assessment of the effect of computerizing noncognitive paper-and-pencil measurements: A test of measurement equivalence. Journal of Applied Psychology, 80: 643-651.

Kiss, H.J., Rodriguez-Lara, I., & Rosa-García, A. (2015). Think Twice Before Running! Bank Runs and Cognitive Abilities. Forthcoming in Journal of Behavioral and Experimental Economics.

Levitt, S.D., & List, J.A. (2007). What do laboratory experiments measuring social preferences reveal about the real world? Journal of Economic Perspectives, 21(2): 153–174.

Liberali, J. M., Reyna, V. F., Furlan, S., Stein, L. M., & Pardo, S. T. (2012). Individual Differences inNumeracy and Cognitive Reflection, with Implications for Biases and Fallacies in Probability Judgment. Journal of Behavioral Decision Making, 25(4): 361–381.

Mau, W.C., & Lynn, R. (2010). Gender differences in homework and test scores in Mathematics, Reading and Science at tenth and twelfth grade. Psychology, Evolution & Gender, 2(2): 119-125.

Moritz, B., Hill, A.V., & Donohue, K. (2013). Individual Differences in the Newsvendor Problem: Behavior and Cognitive Reflection. Journal of Operations Management, 31(1-2): 72-85.

Nagel, R. (1995). Unraveling in Guessing Games: An Experimental Study. American Economic Review,85(5): 1313-26.

Noussair, C.N., Trautmann, S.T., & van de Kuilen, G. (2014). Higher Order Risk Attitudes, Demographics, and Financial Decisions. Review of Economic Studies, 81 (1): 325-355.

Oechssler, J., Roider, A., & Schmitz, P.W. (2009). Cognitive abilities and behavioral biases. Journal of Economic Behavior & Organization, 72(1):147-152.

Paolacci, G., Chandler, J., & Ipeirotis, P.G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5): 411–419.

Peterson, R.A. (2001). On the Use of College Students in Social Science Research: Insights from a Second-Order Meta-analysis. Journal of Consumer Research, 28(3): 450-461.

Peysakhovich, A., & Rand, D.G. (2015). Habits of Virtue: Creating Norms of Cooperation and Defection in the Laboratory. Forthcoming in Management Science.

Ponti, G., & Rodriguez-Lara, I. (2015). Social Preferences and Cognitive Reflection: Evidence from Dictator Game Experiment. Forthcoming in Frontiers in Behavioral Neuroscience.

Riedel, J.A., Nebeker, D.M., & Cooper, B.L. (1988). The Influence of Monetary Incentive on Goal Choice, Goal Commitment, and Task Performance, Organizational Behavior and Human Decision Processes, 42: 155-180.

Scott, W. E., Farh, J.L., & Podsakoff, P.M. (1988). The Effects of ‘Intrinsic’ and ‘Extrinsic’ Reinforcement Contingencies on Task Behavior. Organizational Behavior and Human Decision Processes, 41: 405-425.

Sloman, S.A. (1996). The empirical case for two systems of reasoning. Psychological Bulletin, 119: 3–22.

Stanovich, K.E., & West, R.F. (2000). Individual differences in reasoning: Implications for the rationality debate? Behavioral and Brain Sciences, 23: 645-665.

Toplak, M.E., West, R.F., & Stanovich, K.E. (2011). The Cognitive Reflection Test as a predictor of performance on heuristics and biases tasks. Memory & Cognition, 39: 1275-1289.

Toplak, M.E., West, R.F., & Stanovich, K.E. (2014). Assessing miserly information processing: An expansion of the Cognitive Reflection Test. Thinking & Reasoning, 20(2): 147-168.

Weiss, V. (1986). From memory span to the quantum mechanics of intelligence. Personality and Individual Differences, 7: 737-749.