Regression to the Mean and Selection Bias Checks

Description

Guards against "improvements" that are really just extreme scores drifting back to normal, or samples that were chosen in a biased way.

If you study the worst-performing tasks or the angriest customers, they will usually look better next time even if nothing changed, because extreme scores contain extra noise that does not repeat. This is regression to the mean, and it makes many redesigns look successful. Selection bias arises when the people or data you analyse were chosen by a process related to the outcome, such as only surveying active users or only reviewing sessions that triggered a feedback widget. Bruce and Bruce show how to recognise both; the cure is control groups, random sampling and pre-registration of what will be compared.

Attributes

Loss type
Vestigial
Effort
Hours

Steps

  1. Ask how the sample or items were selected, and whether selection depended on the outcome.
  2. When following up extreme cases, include a comparison group that received no change.
  3. Re-measure baseline twice where possible to estimate natural fluctuation.
  4. Prefer random samples of users or sessions over self-selected ones.
  5. State likely biases explicitly in the report.

Sources

  • Practical Statistics for Data Scientists (Bruce & Bruce)

Also known as

  • Regression to the mean
  • Selection bias audit

Categories

  • Research Operations
  • Research Design & Rigor