Statistical power analysis

Description

Determines the probability that a study will detect an effect of a given size, used to plan sample sizes or to judge what a completed study could have found.

Power is the probability of rejecting the null hypothesis when a specified true effect exists, equal to 1 minus the Type II error rate. It depends on effect size, sample size, alpha, variability and design. Cohen found that psychology studies in the 1960s had a median power of only about 0.46 for medium effects, and underpowered research remains common, producing both missed effects and exaggerated estimates among significant results (the winner's curse). A priori power analysis sets sample size before data collection, typically targeting 80% or 90% power for the smallest effect of interest. Sensitivity analysis asks what effect a fixed sample could detect. Post-hoc 'observed power' is uninformative and should not be used. Rigour comes from basing the effect size on decision relevance or credible prior evidence rather than optimism.

Attributes

Loss type
Vestigial
Effort
Hours

Steps

  1. Define the primary outcome, test and design (for example two-proportion test, two independent groups).
  2. Set the smallest effect size of interest from business relevance or prior evidence, such as a 2 percentage point uplift from a 10% baseline.
  3. Choose alpha (usually 0.05, two-sided) and target power (usually 0.80, or 0.90 for important decisions).
  4. Compute the required sample with appropriate software, adjusting for expected attrition, clustering or multiple comparisons.
  5. If the available sample is fixed, run a sensitivity analysis to state the minimum detectable effect and decide whether the study is worth running.
  6. Document the calculation in the pre-registration or analysis plan, and do not compute observed power after the fact.

Sources

  • Statistics for Dummies series (Rumsey)
  • Designing Experiments & Analyzing Data (Maxwell & Delaney)
  • The Inquiry Series, Ch 41 — Frequentist Inference Done Right
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • Practical Statistics for Data Scientists (Bruce & Bruce)
  • Quantifying the User Experience (Sauro & Lewis)

Also known as

  • Power analysis
  • A priori power analysis
  • Sensitivity analysis

Categories

  • Research Operations
  • Research Design & Rigor