Effect size (Cohen's d, η², ω²)

Description

Quantifies how large a difference or relationship is, independent of sample size, so UX decisions rest on magnitude rather than on statistical significance alone.

Statistical significance depends heavily on sample size; effect size measures magnitude. Cohen's d expresses a mean difference in standard deviation units, with conventional benchmarks of 0.2 small, 0.5 medium and 0.8 large, which Cohen himself cautioned were a last resort. Eta squared (η²) is the proportion of variance explained in ANOVA but is positively biased in samples; omega squared (ω²) corrects this bias and is preferred for population inference. For proportions, absolute differences, relative risk and odds ratios are often more meaningful to stakeholders. In UX, raw unstandardised effects such as seconds saved or percentage points of conversion are frequently the most interpretable. Rigour comes from reporting effect sizes with confidence intervals, choosing the measure that fits the design, and interpreting magnitude against practical criteria.

Attributes

Loss type
Vestigial
Effort
Hours

Steps

  1. Choose the effect size suited to the design: Cohen's d or Hedges' g for two means, ω² or partial ω² for ANOVA, r for correlations, and risk differences or odds ratios for proportions.
  2. Compute the effect size and its 95% confidence interval, using Hedges' g correction for small samples (below about 20 per group).
  3. Also report the unstandardised effect in natural units, such as seconds, percentage points or scale points.
  4. Interpret magnitude against practical benchmarks agreed with stakeholders, such as a minimum of 10 seconds saved or 1 point on SUS, not only Cohen's conventions.
  5. Use the effect size and interval from completed studies to inform power analyses and priors for future studies.
  6. Report effect sizes for non-significant results too, to support meta-analysis and avoid file-drawer bias.

Sources

  • Statistics for Dummies series (Rumsey)
  • Designing Experiments & Analyzing Data (Maxwell & Delaney)
  • The Inquiry Series, Ch 41 — Frequentist Inference Done Right

Also known as

  • Effect size reporting
  • Standardised mean difference
  • Variance explained

Categories

  • Analytics & Behaviour
  • Statistics & Modelling