Effect size calculation

Description

Quantifying how large a difference or relationship is, with a confidence interval, so UX findings convey practical importance rather than just statistical significance.

An effect size expresses magnitude on an interpretable scale: raw differences (seconds saved, percentage-point completion gains), standardised differences (Cohen's d, Hedges' g), variance explained (eta squared, R squared), odds ratios, or rank-based measures (rank-biserial r, Cliff's delta). A p-value conflates effect size with sample size, so large analytics samples make trivial differences 'significant' and small usability samples hide large ones. Rigour comes from choosing the effect size that matches the design (Hedges' g for small samples, d_z for paired designs), always reporting a confidence interval, preferring unstandardised units when they are meaningful to stakeholders, and judging importance against a pre-stated smallest effect of interest rather than Cohen's generic benchmarks.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Before analysis, state the smallest effect size of interest in product terms (e.g. 5 percentage points in completion, 10 seconds in task time).
  2. Choose the effect size matching the design: Hedges' g for independent small samples, d_z or d_av for paired data, odds ratio for binary outcomes, rank-biserial for ordinal tests.
  3. Compute it with a confidence interval (pingouin.compute_effsize with bootstrap CIs, or analytical intervals).
  4. Report the unstandardised difference alongside the standardised value whenever units are meaningful.
  5. Interpret against the pre-stated threshold and prior benchmarks, using Cohen's 0.2, 0.5 and 0.8 only as a last resort.
  6. Use the observed effect and its interval to plan the sample size for follow-up studies.

Sources

  • scipy.stats
  • pingouin
  • mord
  • PyMC
  • bambi
  • statsmodels
  • matplotlib
  • seaborn
  • Cohen, J. (1988) Statistical Power Analysis for the Behavioral Sciences, 2nd ed.
  • Cumming, G. (2012) Understanding the New Statistics
  • Sauro, J. & Lewis, J. R. (2016) Quantifying the User Experience, 2nd ed.
  • The Inquiry Series, C2 — Quantitative UX Analysis in Python — Descriptive to Inferential

Also known as

  • Effect size estimation
  • Standardised mean difference
  • Practical significance
  • Effect size reporting
  • Variance explained
  • Effect size (Cohen's d, η², ω²)

Categories

  • Analytics & Behaviour
  • Statistics & Modelling