Description
Determines the probability that a study will detect an effect of a given size, used to plan sample sizes or to judge what a completed study could have found.
Power is the probability of rejecting the null hypothesis when a specified true effect exists, equal to 1 minus the Type II error rate. It depends on effect size, sample size, alpha, variability and design. Cohen found that psychology studies in the 1960s had a median power of only about 0.46 for medium effects, and underpowered research remains common, producing both missed effects and exaggerated estimates among significant results (the winner's curse). A priori power analysis sets sample size before data collection, typically targeting 80% or 90% power for the smallest effect of interest. Sensitivity analysis asks what effect a fixed sample could detect. Post-hoc 'observed power' is uninformative and should not be used. Rigour comes from basing the effect size on decision relevance or credible prior evidence rather than optimism.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Vestigial
- Goal
- Support a decision
- Stage
- Evaluation, Strategy
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Define the primary outcome, test and design (for example two-proportion test, two independent groups).
- Set the smallest effect size of interest from business relevance or prior evidence, such as a 2 percentage point uplift from a 10% baseline.
- Choose alpha (usually 0.05, two-sided) and target power (usually 0.80, or 0.90 for important decisions).
- Compute the required sample with appropriate software, adjusting for expected attrition, clustering or multiple comparisons.
- If the available sample is fixed, run a sensitivity analysis to state the minimum detectable effect and decide whether the study is worth running.
- Document the calculation in the pre-registration or analysis plan, and do not compute observed power after the fact.
Sources
- Statistics for Dummies series (Rumsey)
- Designing Experiments & Analyzing Data (Maxwell & Delaney)
- The Inquiry Series, Ch 41 — Frequentist Inference Done Right
- Companion Book to Quantifying the User Experience (Lewis & Sauro)
- Practical Statistics for Data Scientists (Bruce & Bruce)
- Quantifying the User Experience (Sauro & Lewis)
Also known as
- Power analysis
- A priori power analysis
- Sensitivity analysis
Categories
- Research Operations
- Research Design & Rigor