Running experiments on Amazon Mechanical Turk
Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechan...
| Published in: | Judgment and Decision Making |
|---|---|
| Main Authors: | , , |
| Format: | Article |
| Language: | English |
| Published: |
Cambridge University Press
2010-08-01
|
| Subjects: | |
| Online Access: | http://journal.sjdm.org/10/10630a/jdm10630a.pdf |
| _version_ | 1851877973198635008 |
|---|---|
| author | Gabriele Paolacci Jesse Chandler Panagiotis G. Ipeirotis |
| author_facet | Gabriele Paolacci Jesse Chandler Panagiotis G. Ipeirotis |
| author_sort | Gabriele Paolacci |
| collection | DOAJ |
| container_title | Judgment and Decision Making |
| description | Although Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool. |
| format | Article |
| id | doaj-art-e43e5221c746446ea9e04c912fa30dbd |
| institution | Directory of Open Access Journals |
| issn | 1930-2975 |
| language | English |
| publishDate | 2010-08-01 |
| publisher | Cambridge University Press |
| record_format | Article |
| spelling | doaj-art-e43e5221c746446ea9e04c912fa30dbd2025-08-19T22:14:22ZengCambridge University PressJudgment and Decision Making1930-29752010-08-0155411419Running experiments on Amazon Mechanical TurkGabriele PaolacciJesse ChandlerPanagiotis G. IpeirotisAlthough Mechanical Turk has recently become popular among social scientists as a source of experimental data, doubts may linger about the quality of data provided by subjects recruited from online labor markets. We address these potential concerns by presenting new demographic data about the Mechanical Turk subject population, reviewing the strengths of Mechanical Turk relative to other online and offline methods of recruiting subjects, and comparing the magnitude of effects obtained using Mechanical Turk and traditional subject pools. We further discuss some additional benefits such as the possibility of longitudinal, cross cultural and prescreening designs, and offer some advice on how to best manage a common subject pool.http://journal.sjdm.org/10/10630a/jdm10630a.pdfexperimentationonline researchNAKeywords |
| spellingShingle | Gabriele Paolacci Jesse Chandler Panagiotis G. Ipeirotis Running experiments on Amazon Mechanical Turk experimentation online researchNAKeywords |
| title | Running experiments on Amazon Mechanical Turk |
| title_full | Running experiments on Amazon Mechanical Turk |
| title_fullStr | Running experiments on Amazon Mechanical Turk |
| title_full_unstemmed | Running experiments on Amazon Mechanical Turk |
| title_short | Running experiments on Amazon Mechanical Turk |
| title_sort | running experiments on amazon mechanical turk |
| topic | experimentation online researchNAKeywords |
| url | http://journal.sjdm.org/10/10630a/jdm10630a.pdf |
| work_keys_str_mv | AT gabrielepaolacci runningexperimentsonamazonmechanicalturk AT jessechandler runningexperimentsonamazonmechanicalturk AT panagiotisgipeirotis runningexperimentsonamazonmechanicalturk |