Description
Reducing and correcting selection bias in opt-in samples through recruitment design, sample matching and propensity weighting against a small probability reference survey.
Most UX surveys use opt-in samples: panels, intercepts, social recruitment. This method treats their selection bias as something to design against and adjust for. At recruitment, diverse channels and quotas reduce bias; sample matching draws a target sample from a frame and selects the closest panel member for each target case. After fieldwork, post-stratification, raking or calibration align known margins. The stronger correction runs a small probability reference survey in the same mode with shared questions, fits a logistic model of membership (opt-in versus reference) and weights opt-in cases by inverse propensity. Results are then validated against external benchmarks and reported as indications rather than population estimates. The key limitation is explicit: adjustment removes only the bias explained by the model variables, and weighting increases variance.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Siloed
- Stage
- Discovery, Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Spread recruitment across several channels and set quotas on variables known to relate to the outcomes.
- Optionally draw a target sample from a frame and match each target case to the closest panel member on shared covariates.
- After fieldwork, weight to known margins by post-stratification, raking or calibration.
- Run a small probability reference survey in the same mode, with the covariates and some outcome questions in common.
- Fit a logistic model of membership (opt-in versus reference) and weight opt-in cases by inverse propensity, trimming extreme weights.
- Validate adjusted estimates against external benchmarks and report them as indications with the basis of adjustment stated.
Sources
- The SAGE Handbook of Survey Methodology (Wolf et al., 2016)
Also known as
- Online panel bias adjustment
- Reference survey propensity weighting
- Sample matching
- Opt-in sample adjustment
- Non-probability (online panel) sampling
Categories
- Analytics & Behaviour
- Survey & Psychometrics