Description
Run controlled experiments in the browser with large samples of online or paid crowd participants, using designed-in quality checks, fair pay and dropout analysis.
Browser-based platforms and crowd marketplaces (MTurk, Prolific, CloudResearch) let researchers run experiments with hundreds of participants in days. Results usually match the lab, even for perception and reaction-time tasks: Heer and Bostock replicated lab perception studies on MTurk with wider variance but the same conclusions at one-sixth of the cost. Quality must be designed in: questions with known answers, making accurate answers as easy as careless ones, screening for rushed or repeated responses, redundancy and agreement models, instruction checks, and logging tab-hidden focus loss. Instructions must stand alone and be pictorial where possible; sessions should be short, since dropout can reach 69%, and dropout must be analysed by condition. Pairing with a small lab study corroborates both: Meyer and Bederson's 221 online participants matched 18 in the lab. Fair pay and treatment of workers is an ethical requirement.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Active
- Stage
- Design, Evaluation
- Data
- Quantitative
- Effort
- Days
Steps
- Choose a platform, host and recruitment source (crowd marketplace or own panel), and decide whether a custom web app is needed for logging.
- Write stand-alone, pictorial instructions with success criteria, and add an instruction or comprehension check.
- Build in quality controls: known-answer questions, attention checks, timing and repetition screens, and logging of focus loss and response times.
- Set fair pay based on realistic completion time and keep sessions short; pilot tasks and software with a small worker sample.
- Run the study, monitor quality and engage worker communities to resolve problems fairly.
- Analyse dropout by condition, report exclusions and which participants each analysis includes.
- Where possible, run the same study with a small lab sample and compare result patterns; investigate discrepancies.
Sources
- Research Methods in HCI (Lazar et al., 2017)
- Interaction Design: Beyond Human-Computer Interaction (Preece et al., 2019)
- Cambridge Handbook of Research Methods (Nichols & Edlund, 2023)
- Understanding Audiences via Analytics (Jansen et al., 2024)
- The Practice of Social Research (Babbie, 2013)
Also known as
- Crowdsourced HCI experiments with quality controls
- Mechanical Turk studies
- Crowdsourced user studies
- Crowdsourced online experiments
- Online experiment practices
- Web experiments
- Browser-based experiments
- Remote experiments
- Crowdsourced data collection with quality controls
- Crowd-worker studies
- Paired lab and online replication
- Lab-online pairing
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference