Interactive experiments in Toloka

Chapkovski, Philipp (2022): Interactive experiments in Toloka.

Abstract

The popularity of online behavioral experiments grew steadily even before the COVID-19 pandemic. With the start of lockdowns, online studies were often the only available option for the behavioral economists, sociologists and political scientists. The usage of most well-known platforms such as mTurk was so intensive that it harmed the quality of data. But even before the pandemics-induced quality crisis, online studies were limited in scope, since real-time interactions between participants were hard to achieve due to the large proportion of drop-outs and issues with creating stable groups. Using the crowdsourcing platform Toloka, we successfully ran several multi-round interactive experiments. Toloka’s large online audience, relatively low exposure of participants to sociological surveys and behavioral studies, and a convenient application programming interface makes it a perfect tool to run behavioral studies that require real-time interactions of participants.

• Item Type: MPRA Paper
• Language: English
• Keywords: Crowdsourcing, mTurk, online research, survey research
• Subjects: B - History of Economic Thought, Methodology, and Heterodox Approaches > B4 - Economic Methodology > B41 - Economic Methodology; C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs > C81 - Methodology for Collecting, Estimating, and Organizing Microeconomic Data ; Data Access; C - Mathematical and Quantitative Methods > C8 - Data Collection and Data Estimation Methodology ; Computer Programs > C88 - Other Computer Software; C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C90 - General; C - Mathematical and Quantitative Methods > C9 - Design of Experiments > C92 - Laboratory, Group Behavior

References

Abeler, Johannes, Anke Becker, and Armin Falk. 2014. “Representative Evidence on Lying Costs.” Journal of Public Economics 113 (Supplement C): 96–104.

Abeler, Johannes, Armin Falk, Lorenz Goette, and David Huffman. 2011. “Reference Points and Effort Provision.” American Economic Review 101 (2): 470–92.

Amir, Ofra, David G. Rand, and Ya’akov Kobi Gal. 2012. “Economic Games on the Internet: The Effect of $1 Stakes.” PLOS ONE 7 (2): e31461.

Andreoni, James. 1988. “Why Free Ride?: Strategies and Learning in Public Goods Experiments.” Journal of Public Economics 37 (3): 291–304.

Arechar, Antonio A., Simon Gächter, and Lucas Molleman. 2018. “Conducting Interactive Experiments Online.” Experimental Economics 21 (1): 99–131.

Arechar, Antonio A., Gordon T. Kraft-Todd, and David G. Rand. 2017. “Turking Overtime: How Participant Characteristics and Behavior Vary over Time and Day on Amazon Mechanical Turk.” Journal of the Economic Science Association 3 (1): 1–11.

Barends, Ard J., and Reinout E. de Vries. 2019. “Noncompliant Responding: Comparing Exclusion Criteria in MTurk Personality Research to Improve Data Quality.” Personality and Individual Differences 143: 84–89.

Battaglini, Marco, Rebecca B. Morton, and Thomas R. Palfrey. 2010. “The Swing Voter’s Curse in the Laboratory.” The Review of Economic Studies 77 (1): 61–89.

Benndorf, Volker, Claudia Moellers, and Hans-Theo Normann. 2017. “Experienced vs. Inexperienced Participants in the Lab: Do They Behave Differently?” Journal of the Economic Science Association 3 (1): 12–25.

Bergh, Andreas, and Philipp C Wichardt. 2022. “Mine or Ours? Unintended Framing Effects in Dictator Games.” Rationality and Society 34 (1): 78–95.

Brandts, J., and G. Charness. 2011. “The Strategy versus the Direct-Response Method: A First Survey of Experimental Comparisons.” Experimental Economics 14 (3): 375–98.

Cason, Timothy N., Daniel Friedman, and Ed Hopkins. 2014. “Cycles and Instability in a Rock–Paper–Scissors Population Game: A Continuous Time Experiment.” Review of Economic Studies 81 (1): 112–36.

Chandler, Jesse, Pam Mueller, and Gabriele Paolacci. 2014. “Nonnaïveté among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers.” Behavior Research Methods 46 (1): 112–30.

Charness, Gary, Uri Gneezy, and Brianna Halladay. 2016. “Experimental Methods: Pay One or Pay All.” Journal of Economic Behavior & Organization 131: 141–50.

Chen, Daniel L., Martin Schonger, and Chris Wickens. 2016. “OTree—An Open-Source Platform for Laboratory, Online, and Field Experiments.” Journal of Behavioral and Experimental Finance 9: 88–97.

Cherry, Todd L., Peter Frykblom, and Jason F. Shogren. 2002. “Hardnose the Dictator.” American Economic Review 92 (4): 1218–21.

Chmielewski, Michael, and Sarah C. Kucker. 2020. “An MTurk Crisis? Shifts in Data Quality and the Impact on Study Results.” Social Psychological and Personality Science 11 (4): 464–73.

Cooper, David J., and Hanming Fang. 2008. “Understanding Overbidding in Second Price Auctions: An Experimental Study.” The Economic Journal 118 (532): 1572–95.

Difallah, Djellel, Elena Filatova, and Panos Ipeirotis. 2018. “Demographics and Dynamics of Mechanical Turk Workers.” In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, 135–43. WSDM ’18. New York, NY, USA: Association for Computing Machinery.

Dyson, Benjamin James, Jonathan Michael Paul Wilbiks, Raj Sandhu, Georgios Papanicolaou, and Jaimie Lintag. 2016. “Negative Outcomes Evoke Cyclic Irrational Decisions in Rock, Paper, Scissors.” Scientific Reports 6 (1): 20479.

Engel, Christoph. 2011. “Dictator Games: A Meta Study.” Experimental Economics 14 (4): 583–610. Fehr, Ernst, and Simon Gächter. 1999. “Cooperation and Punishment in Public Goods Experiments.” Institute for Empirical Research in Economics Working Paper, no. 10.

Fischbacher, Urs, and Franziska Föllmi-Heusi. 2013. “Lies in Disguise—an Experimental Study on Cheating.” Journal of the European Economic Association 11 (3): 525–47.

Fischbacher, Urs, Simon Gächter, and Simone Quercia. 2012. “The Behavioral Validity of the Strategy Method in Public Good Experiments.” Journal of Economic Psychology 33 (4): 897–913.

Gächter, Simon, and Benedikt Herrmann. 2011. “The Limits of Self-Governance When Cooperators Get Punished: Experimental Evidence from Urban and Rural Russia.” European Economic Review 55 (2): 193–210.

Haerpfer, Christian, Ronald Inglehart, A. Moreno, Christian Welzel, K. Kizilova, J. Diez-Medrano, Marta Lagos, Pippa Norris, Eduar Ponarin, and Bi Puranen. 2020. “World Values Survey: Round Seven–Country-Pooled Datafile.” Madrid, Spain & Vienna, Austria: JD Systems Institute & WVSA Secretariat.

Heinz, Matthias, Steffen Juranek, and Holger A. Rau. 2012. “Do Women Behave More Reciprocally than Men? Gender Differences in Real Effort Dictator Games.” Journal of Economic Behavior & Organization, Gender Differences in Risk Aversion and Competition, 83 (1): 105–10.

Henrich, Joseph, Steven J. Heine, and Ara Norenzayan. 2010. “The Weirdest People in the World?” Behavioral and Brain Sciences 33 (2–3): 61–83.

Herrmann, Benedikt, Christian Thöni, and Simon Gächter. 2008. “Antisocial Punishment across Societies.” Science 319 (5868): 1362–67.

Hoffman, Moshe, Sigrid Suetens, Uri Gneezy, and Martin A. Nowak. 2015. “An Experimental Investigation of Evolutionary Dynamics in the Rock-Paper-Scissors Game.” Scientific Reports 5 (1): 1–7.

Hugh-Jones, David. 2016. “Honesty, Beliefs about Honesty, and Economic Growth in 15 Countries.” Journal of Economic Behavior & Organization 127: 99–114.

Isaac, R. M., and J. M. Walker. 1988. “Group-Size Effects In Public-Goods Provision - The Voluntary Contributions Mechanism.” Quarterly Journal Of Economics 103 (1): 179–99.

Kennedy, Ryan, Scott Clifford, Tyler Burleigh, Philip D. Waggoner, Ryan Jewell, and Nicholas JG Winter. 2020. “The Shape of and Solutions to the MTurk Quality Crisis.” Political Science Research and Methods 8 (4): 614–29.

Krupka, Erin L., and Roberto A. Weber. 2013. “Identifying Social Norms Using Coordination Games: Why Does Dictator Game Sharing Vary?” Journal of the European Economic Association 11 (3): 495–524.

Lee, Yun Shin, Yong Won Seo, and Enno Siemsen. 2018. “Running Behavioral Operations Experiments Using Amazon’s Mechanical Turk.” Production and Operations Management 27 (5): 973–89.

Levada. 2021. “Koronavirus, vakcinacija, QR-kody.” July 12, 2021. https://www.levada.ru/2021/12/07/koronavirus-vaktsinatsiya-qr-kody/.

Nagel, Rosemarie. 1995. “Unraveling in Guessing Games: An Experimental Study.” The American Economic Review 85 (5): 1313–26.

Necka, Elizabeth A., Stephanie Cacioppo, Greg J. Norman, and John T. Cacioppo. 2016. “Measuring the Prevalence of Problematic Respondent Behaviors among MTurk, Campus, and Community Participants.” PLOS ONE 11 (6): e0157732.

Niederle, Muriel, and Lise Vesterlund. 2007. “Do Women Shy Away from Competition? Do Men Compete Too Much?” The Quarterly Journal of Economics 122 (3): 1067–1101.

Palan, Stefan, and Christian Schitter. 2018. “Prolific. Ac—A Subject Pool for Online Experiments.” Journal of Behavioral and Experimental Finance 17: 22–27.

Peer, Eyal, Laura Brandimarte, Sonam Samat, and Alessandro Acquisti. 2017. “Beyond the Turk: Alternative Platforms for Crowdsourcing Behavioral Research.” Journal of Experimental Social Psychology 70: 153–63.

Selten, Reinhard. 1967. “Die Strategiemethode Zur Erforschung Des Eingeschränkt Rationalen Verhaltens Im Rahmen Eines Oligopolexperiments.” In Beiträge Zur Experimentellen Wirtschaftsforschung, edited by Ernst Sauermann, 136–68. Tübingen: JCB Mohr (Paul Siebeck) Tübingen.

Snowberg, Erik, and Leeat Yariv. 2021. “Testing the Waters: Behavior across Participant Pools.” American Economic Review 111 (2): 687–719.

Stewart, Neil, Christoph Ungemach, Adam JL Harris, Daniel M. Bartels, Ben R. Newell, Gabriele Paolacci, and Jesse Chandler. 2015. “The Average Laboratory Samples a Population of 7,300 Amazon Mechanical Turk Workers.” Judgment and Decision Making 10 (5): 479–91.

Wang, Zhijian, Bin Xu, and Hai-Jun Zhou. 2014. “Social Cycling and Conditional Responses in the Rock-Paper-Scissors Game.” Scientific Reports 4 (1): 5830.