Non-Probability Sample Adjustment (Sample Matching and Propensity Weighting)

Description

Reducing and correcting selection bias in opt-in samples through recruitment design, sample matching and propensity weighting against a small probability reference survey.

Most UX surveys use opt-in samples: panels, intercepts, social recruitment. This method treats their selection bias as something to design against and adjust for. At recruitment, diverse channels and quotas reduce bias; sample matching draws a target sample from a frame and selects the closest panel member for each target case. After fieldwork, post-stratification, raking or calibration align known margins. The stronger correction runs a small probability reference survey in the same mode with shared questions, fits a logistic model of membership (opt-in versus reference) and weights opt-in cases by inverse propensity. Results are then validated against external benchmarks and reported as indications rather than population estimates. The key limitation is explicit: adjustment removes only the bias explained by the model variables, and weighting increases variance.

Attributes

Loss type
Siloed
Effort
Weeks

Steps

  1. Spread recruitment across several channels and set quotas on variables known to relate to the outcomes.
  2. Optionally draw a target sample from a frame and match each target case to the closest panel member on shared covariates.
  3. After fieldwork, weight to known margins by post-stratification, raking or calibration.
  4. Run a small probability reference survey in the same mode, with the covariates and some outcome questions in common.
  5. Fit a logistic model of membership (opt-in versus reference) and weight opt-in cases by inverse propensity, trimming extreme weights.
  6. Validate adjusted estimates against external benchmarks and report them as indications with the basis of adjustment stated.

Sources

  • The SAGE Handbook of Survey Methodology (Wolf et al., 2016)

Also known as

  • Online panel bias adjustment
  • Reference survey propensity weighting
  • Sample matching
  • Opt-in sample adjustment
  • Non-probability (online panel) sampling

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics