Conducting Online Research on Amazon Mechanical Turk and Beyond

Leib Litman received a PhD in experimental psychology from the City University of New York. His early research was in the area of implicit learning, focusing on the relationship between conscious and unconscious information processing. After receiving his PhD, he was a postdoctoral fellow and research scientist in cognitive neuroscience at New York University, where he studied episodic memory using fMRI. Leib's current methodological research interests are in the area of online data collection, especially in the application of online data collection tools to complex research designs, such as dyadic studies, experience sampling, intensive longitudinal designs, qualitative video and focus group interviews, and global data collection. Leib is currently an associate professor of psychology at Lander College in Queens, New York, and director of research at CloudResearch.com (formerly TurkPrime). Leib lives in Riverdale, New York, with his wife, Sarah, and their two girls, Bella and Rebecca. Jonathan Robinson has been creating software since he was 12 and started his first software company with a few high school friends at age 15. He continued to develop his software skills and apply his expertise to more complex problems as he completed his BA at Queens College, and his MS and PhD with a dissertation in the field of robotic vision and digital curve fitting while at the City University of New York. Jonathan conceived of using Amazon's Mechanical Turk for participants before there was public academic discussion of its application to research. He designed and implemented the core CloudResearch.com (previously TurkPrime) system together with his coauthor, Leib Litman. CloudResearch was the first cloud-based participant management system that successfully adapted Mechanical Turk for research by designing the features researchers need and managing the complexity that may get in the way of collecting high-quality data. Jonathan is the co-CEO and chief technology officer of CloudResearch. He is also currently an associate professor and chair of computer science at Lander College, a division of Touro College. In that capacity, he mentors, inspires, and educates the next generation of software engineers and introduces them to his passion for melding cutting-edge technologies with solving real-world problems in the areas of academic and corporate research, digital image computation, approximation, and web application engineering.

Chapter 1: Introduction A Scientific Revolution in the Making A Brief History of Online Research in the Social and Behavioral Sciences: From HTML 2.0 to Mechanical Turk The Use of Online Samples in Applied Behavioral Research Amazon Mechanical Turk Chapter 2: The Mechanical Turk Ecosystem - Leib Litman, Cheskie Rosenzweig, Jonathan Robinson How Quality is Maintained Reputation Mechanism Selectively Recruiting Specific Workers Protections for Workers Communicating with Workers A Worker's Perspective Worker Communities Chapter 3: Conducting a Study on Mechanical Turk Sample Project Setting up a Requester Account on Mechanical Turk Creating a HIT The 'Design Layout' tab Monitoring Progress on the Requester's Dashboard When a Worker Runs Out of Time Sample Study Results Conducting Follow-up Studies Using Requester-Issued Qualifications Appendix A: Checklist for best practices of setting up a Mechanical Turk HIT Chapter 4: API and Third Party Apps Third Party API-based Platforms Common Uses for API Scripts and Third Party API-based Apps TurkPrime Chapter 5: Data Quality Issues on MTurk - Jesse Chandler, Gabriele Paolacci, David Hauser Defining and Measuring Data Quality Measuring Individual Participant Data Quality Causes of and Cures for Poor Data Quality Concluding Thoughts Chapter 6: Who are the Mechanical Turk Workers? Sources of Data Location of Workers in the US Demographics of Mechanical Turk Chapter 7: Sampling Mechanical Turk Workers: Problems and Solutions Sampling on Mechanical Turk Sources of Sampling Bias The Problem of Superworkers Time-of-day Effects Pay Rate Dropout Sampling Best Practices Chapter 8: Data Representativeness of Mechanical Turk Samples Representativeness, surveys, and survey sampling The methodology of survey sampling Mechanical Turk as a sampling frame The fit-for-purpose framework Chapter Overview: Chapter 9: Conducting Longitudinal Research on Amazon Mechanical Turk Why Longitudinal Research? Retention, Longitudinal Research, and MTurk Case Studies Best practices for longitudinal research Chapter 10: Beyond Mechanical Turk: Using Online Market Research Platforms Limitations of MTurk Online probability-based panels Online market research platforms Overall comparisons between Mechanical Turk and market research platforms Chapter 11: Conducting Ethical Online Research: A Data-Driven Approach Historical background Risk of harm in online research Research on sensitive topics A deeper dive into controversial and complex issues Economics of Mechanical Turk: Considerations for setting wages Setting wages: Considerations of ethics and methodology Considerations for rejecting, blocking and disqualifying workers. Practical advice for requester/worker interactions Anonymity Appendix A

"This is the best practical guide I have seen written in any methodologically written book!" -- Michaela Porubanova "Great overview of literature developing in several fields; it's hugely valuable to have all of this in one place." -- Kevin Munger "They cover the whole gamut of history/background, practical concerns and scholarly concerns." -- See-yeon Hwang "I am looking forward to being able to hand this to my students and know they'll be ready to get started on MTurk and really understand the whole system (i.e., not just the mechanics of getting data, but really "getting" the culture and so on). I would probably also find myself citing this book as it discusses many of the issues I personally research." -- Alice M. Brawley