Online and crowdsourced experiments with quality controls

Description

Run controlled experiments in the browser with large samples of online or paid crowd participants, using designed-in quality checks, fair pay and dropout analysis.

Browser-based platforms and crowd marketplaces (MTurk, Prolific, CloudResearch) let researchers run experiments with hundreds of participants in days. Results usually match the lab, even for perception and reaction-time tasks: Heer and Bostock replicated lab perception studies on MTurk with wider variance but the same conclusions at one-sixth of the cost. Quality must be designed in: questions with known answers, making accurate answers as easy as careless ones, screening for rushed or repeated responses, redundancy and agreement models, instruction checks, and logging tab-hidden focus loss. Instructions must stand alone and be pictorial where possible; sessions should be short, since dropout can reach 69%, and dropout must be analysed by condition. Pairing with a small lab study corroborates both: Meyer and Bederson's 221 online participants matched 18 in the lab. Fair pay and treatment of workers is an ethical requirement.

Attributes

Loss type
Active
Effort
Days

Steps

  1. Choose a platform, host and recruitment source (crowd marketplace or own panel), and decide whether a custom web app is needed for logging.
  2. Write stand-alone, pictorial instructions with success criteria, and add an instruction or comprehension check.
  3. Build in quality controls: known-answer questions, attention checks, timing and repetition screens, and logging of focus loss and response times.
  4. Set fair pay based on realistic completion time and keep sessions short; pilot tasks and software with a small worker sample.
  5. Run the study, monitor quality and engage worker communities to resolve problems fairly.
  6. Analyse dropout by condition, report exclusions and which participants each analysis includes.
  7. Where possible, run the same study with a small lab sample and compare result patterns; investigate discrepancies.

Sources

  • Research Methods in HCI (Lazar et al., 2017)
  • Interaction Design: Beyond Human-Computer Interaction (Preece et al., 2019)
  • Cambridge Handbook of Research Methods (Nichols & Edlund, 2023)
  • Understanding Audiences via Analytics (Jansen et al., 2024)
  • The Practice of Social Research (Babbie, 2013)

Also known as

  • Crowdsourced HCI experiments with quality controls
  • Mechanical Turk studies
  • Crowdsourced user studies
  • Crowdsourced online experiments
  • Online experiment practices
  • Web experiments
  • Browser-based experiments
  • Remote experiments
  • Crowdsourced data collection with quality controls
  • Crowd-worker studies
  • Paired lab and online replication
  • Lab-online pairing

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference