Systems Perspective of Amazon Mechanical Turk for Organizational Research: Review and Recommendations

• Main Authors: Melissa G. Keith, Louis Tay, Peter D. Harms
• Format: Article
• Language: English
• Published: Frontiers Media S.A. 2017-08-01
• Series: Frontiers in Psychology
• Subjects: mechanical turk; sampling; review; meta-analysis; research design
• Online Access: http://journal.frontiersin.org/article/10.3389/fpsyg.2017.01359/full

Amazon Mechanical Turk (MTurk) is becoming a prevalent source of quick and cost effective data for organizational research, but there are questions about the appropriateness of the platform for organizational research. To answer these questions, we conducted an integrative review based on 75 papers evaluating the MTurk platform and 250 MTurk samples used in organizational research. This integrative review provides four contributions: (1) we analyze the trends associated with the use of MTurk samples in organizational research; (2) we develop a systems perspective (recruitment system, selection system, and work management system) to synthesize and organize the key factors influencing data collected on MTurk that may affect generalizability and data quality; (3) within each factor, we also use available MTurk samples from the organizational literature to analyze key issues (e.g., sample characteristics, use of attention checks, payment); and (4) based on our review, we provide specific recommendations and a checklist for data reporting in order to improve data transparency and enable further research on this issue.