Amazon Mechanical Turk Workers Can Provide Consistent and Economically Meaningful Data

Johnson, David and Ryan, John (2018): Amazon Mechanical Turk Workers Can Provide Consistent and Economically Meaningful Data.

Abstract

We explore the consistency of the characteristics of individuals who participate in studies posted on Amazon Mechanical Turk (AMT). The primary individuals analyzed in this study are subjects who participated in at least two of eleven experiments that were run on AMT between September of 2012 to January of 2018. We demonstrate subjects consistently report a series of demographic and personality characteristics. Further, subjective willingness to take risk is found to be significantly correlated with decisions made in a simple lottery experiment with real stakes - even when the subjective risk measure is reported months, sometimes years, in the past. This suggests the quality of data obtained via AMT is not significantly harmed by the lack of control over the conditions under which the responses are recorded.

• Item Type: MPRA Paper
• Language: English
• Keywords: Online Experiment; Risk; Consistency; Amazon Mechanical Turk; Experiment
• Subjects: C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs > C81 - Methodology for Collecting, Estimating, and Organizing Microeconomic Data ; Data Access; C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs > C89 - Other; C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C90 - General; C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C99 - Other

References

Arechar, Antonio A, Gordon T Kraft-Todd, and David G Rand (2017) ‘Turking Overtime: How Participant Characteristics and Behavior Vary Over Time and Day on Amazon Mechanical Turk.’ Journal of the Economic Science Association 3(1), 1–11

Behrend, Tara S, David J Sharek, Adam W Meade, and Eric N Wiebe (2011) ‘The Viability of Crowdsourcing for Survey Research.’ Behavior research methods 43(3), 800

Benjamini, Yoav, and Yosef Hochberg (1995) ‘Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.’ Journal of the Royal Statistical Society, Series B (Methodological) 57(1), 289–300

Berinsky, Adam J, Gregory A Huber, and Gabriel S Lenz (2012) ‘Evaluating Online Labor Markets for Experimental Research: Amazon.com’s Mechanical Turk.’ Political Analysis 20(3), 351–368

Buhrmester, Michael, Tracy Kwang, and Samuel D Gosling (2011) ‘Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data?’ Perspectives on Psychological Science 6(1), 3–5

Carlin, Ryan E., and Gregory J. Love (2013) ‘The Politics of Interpersonal Trust and Reciprocity: An Experimental Approach.’ Political Behavior 35(1), 43–63

Clifford, Scott (2014) ‘Linking issue stances and trait inferences: a theory of moral exemplification.’ The Journal of Politics 76(3), 698–710

Clifford, Scott, and Ben Gaskins (2016) ‘Trust Me, I Believe in God: Candidate Religiousness as a Signal of Trustworthiness.’ American Politics Research 44(6), 1066–1097

Converse, Philip E. ‘Information flow and the stability of partisan attitudes.’ Public Opinion Quarterly 26(4), 578–599

Dave, Chetan, Catherine C. Eckel, Cathleen A. Johnson, and Christian Rojas (2010) ‘Eliciting Risk Preferences: When is Simple Better?’ Journal of Risk and Uncertainty 41(3), 219–243

Del Ponte, Alessandro, Andrew W. Delton, Reuben Kline, and Nicholas A. Seltzer (2017) ‘Passing It Along: Experiments on Creating the Negative Externalities of Climate Change.’ Journal of Politics 79(4), 1444–1448

Dohmen, Thomas, Armin Falk, David Huffman, Uwe Sunde, Jürgen Schupp, and Gert G Wagner (2011) ‘Individual Risk Attitudes: Measurement, Determinants, and Behavioral Consequences.’ Journal of the European Economic Association 9(3), 522–550

Farrington, David P. (1998) ‘Predictors, Causes, and Correlates of Male Youth Violence.’ Crime and Justice 24, 421–475

Flores, Andrew R, Taylor NT Brown, and Jody Herman (2016) Race and Ethnicity of Adults Who Identify as Transgender in the United States (Williams Institute, UCLA School of Law)

Fowler, James H., and Cindy D. Kam (2007) ‘Beyond the Self: Social Identity, Altruism, and Political Participation.’ The Journal of Politics 69(3), 813–827

Gibson, John, and David Johnson (2018) ‘Assessing the Stability of Risk Preferences Online.’ Technical Report, University of Central Missouri

Goodman, Joseph K., Cynthia E. Cryder, and Amar Cheema (2013) ‘Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples.’ Journal of Behavioral Decision Making 26(3), 213–224

Goodman, William K, Ashley M Geiger, and Jutta M Wolf (2016) ‘Differential Links between Leisure Activities and Depressive Symptoms in Unemployed Individuals.’ Journal of Clinical Psychology 72(1), 70–78

Hatemi, Peter K., John R. Alford, John R. Hibbing, Nicholas G. Martin, and Lindon J. Eaves (2009) ‘Is There a ’Party’ in Your Genes?’ Political Psychology (3), 584–600

Holt, Charles A, and Susan K Laury (2002) ‘Risk Aversion and Incentive Effects.’ American Economic review 92(5), 1644–1655

Horton, John J, David G Rand, and Richard J Zeckhauser (2011) ‘The Online Laboratory:Conducting Experiments in a Real Labor Market.’ Experimental Economics 14(3), 399–425

Huff, Connor, and Dustin Tingley (2015) ‘“Who are These People?” Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents.’ Research & Politics 2(3), 2053168015604648

Ipeirotis, Panagiotis G (2010) ‘Demographics of Mechanical Turk.’

Johnson, David B, and Matthew D Webb (2016) ‘Decision Making with Risky, Rival Outcomes: Theory and Evidence.’ Technical Report, Carleton University, Department of Economics

Krupnikov, Yanna, and Adam Seth Levine (2014) ‘Cross-Sample Comparisons and External Validity.’ Journal of Experimental Political Science 1(1), 59–80

Levay, Kevin E, Jeremy Freese, and James N Druckman (2016) ‘The Demographic and Political Composition of Mechanical Turk Samples.’ SAGE Open 6(1), 2158244016636433

Lopez, Jesse, and D. Sunshine Hillygus (2018) ‘Why So Serious?: Survey Trolls and Misinformation.’ Available at SSRN: https://ssrn.com/abstract=3131087 or http://dx.doi.org/10.2139/ssrn.3131087

McAdams, Dan P., Kathrin J. Hanek, and Joseph G. Dadabo (2013) ‘Themes of Self-Regulation and Self-Exploration in the Life Stories of Religious American Conservatives and Liberals.’ Political Pyschology 34(2), 201–219

McDermott, Rose, Chris Dawes, Elizabeth Prom-Wormley, Lindon Eaves, and Peter K. Hatemi (2013) ‘MAOA and Aggression: A Gene–Environment Interaction in Two Populations.’ The Journal of Conflict Resolution 57(6), 1043–1064

Milita, Kerri, Elizabeth N. Simas, John Barry Ryan, and Yanna Krupnikov (2017) ‘The Effects of Ambiguous Rhetoric in Congressional Elections.’ Electoral Studies 46(April), 48–63

Mullinix, Kevin J., Thomas J. Leeper, James N. Druckman, and Jeremy Freese (2015) ‘The Generalizability of Survey Experiments.’ Journal of Experimental Political Science 2(2), 109–138

Paolacci, Gabriele, Jesse Chandler, and Panagiotis G Ipeirotis (2010) ‘Running Experiments on Amazon Mechanical Turk.’

Richter, David, Julia Roher, Maria Metzing, Wiebke Nestler, MichaelWeinhardt, and Jürgen Schupp (2017) ‘SOEP Scales Manual (updated for SOEP-Core v32. 1).’ Technical Report, SOEP Survey Papers

Stanford, Matthew S, Charles W Mathias, Donald M Dougherty, Sarah L Lake, Nathaniel E Anderson, and Jim H Patton (2009) ‘Fifty Years of the Barratt Impulsiveness Scale: AnUpdate and Review.’ Personality and Individual Differences 47(5), 385–395

Verkoeijen, Peter P. J. L., and Samantha Bouwmeester (2014) ‘Does Intuition Cause Cooperation?’ PloS ONE. https://doi.org/10.1371/journal.pone.0096654

Zaller, John (1992) The Nature and Origins of Mass Opinion (New York: Cambridge University Press)