Simpson's paradox and composition-shift check

Description

Before trusting a pooled A/B result or metric trend, checks whether segment shares differ between arms or over time and whether within-segment effects point the other way.

A pooled comparison can reverse within every segment when segments are mixed in different proportions. In Crook and colleagues' A/B example, browsers were sampled at different rates, so treatment won overall but lost in every browser. In Chapman and Rodden's tracking example, pooled customer satisfaction fell about 1.5 points a year, yet the US was flat and Germany rising; the decline came from Germany, with lower ratings, making up a growing share of respondents. The check lists segments that could relate to both allocation or sampling and the outcome, compares their shares across arms or periods, estimates effects within segments, and fits models with segment main effects and segment-by-time or segment-by-treatment interactions. If composition changed, report stratified or reweighted estimates. Only stratify on variables fixed before treatment, and limit slicing to avoid false positives.

Attributes

Loss type
Active
Effort
Hours

Steps

  1. List segments that could affect both allocation or sampling and the outcome: browser, platform, country, plan, acquisition channel, period.
  2. Check that the treatment share is equal across segments and periods (a sample ratio check), or plot each segment's share of respondents over time for tracked metrics.
  3. Compute the effect or trend within each main segment as well as overall, with confidence intervals.
  4. Fit a model with segment main effects and segment-by-treatment or segment-by-time interactions.
  5. If within-segment results disagree with the pooled result, report stratified or reweighted estimates and explain the composition shift.
  6. Fix allocation or sampling so future runs keep segment shares constant, and use qualitative data to judge whether segment level differences are real or scale-use effects.

Sources

  • Designing the User Interface (Shneiderman et al., 2018)
  • Quantitative User Experience Research (Chapman & Rodden, 2023)
  • An Introduction to Categorical Data Analysis (Agresti, 1996)
  • Probability and Statistics (DeGroot & Schervish, 2012)
  • Intermediate Statistics for Dummies (Rumsey, 2007)

Also known as

  • Simpson's paradox check in experiments
  • Simpson's paradox
  • Aggregation reversal check
  • Segment-level consistency check
  • Composition-shift check for tracked metrics
  • Mix-shift analysis
  • Stratified trend analysis

Categories

  • Analytics & Behaviour
  • Statistics & Modelling