Controlling for Prior Experience (Weighted t-Test)

Description

Adjusts competitive comparisons so a product is not rewarded just because more of its testers already used it.

People who already use a product rate it higher and perform better on it. If one product's sample contains more existing users than another's, comparisons are biased. Sauro's approach weights each group so that the proportion of experienced and new users matches across products (or a target mix), then runs a weighted t-test. This is a simple form of post-stratification. It is especially important in competitive benchmarks recruited from general panels, where a market leader will naturally have more prior users.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Hours

Steps

  1. Record prior experience with each product (for example: never, under a year, over a year).
  2. Choose a target mix of experience levels, such as the overall sample proportions.
  3. Compute weights for each participant so each product's sample matches the target mix.
  4. Run a weighted t-test or weighted regression on the metric of interest.
  5. Report unweighted and weighted results side by side.

Sources

  • Benchmarking the User Experience (Sauro)

Also known as

  • Weighted t-test
  • Experience-weighted comparison

Categories

  • Analytics & Behaviour
  • Statistics & Modelling