Surrogate (driver/leading) metric derivation and validation

Description

Find early, measurable behaviours that predict a slow or off-product outcome, then validate causally that moving them moves the outcome, so experiments can be judged in days.

Key outcomes such as retention, purchases of a 13-week course credential, or real-world behaviour change move too slowly or happen outside the product. Teams therefore use surrogate metrics, but often stop at correlation ('seven friends in ten days'). The rigorous version has two halves. Derivation: start from the key outcome, trace the behaviours that must precede it, use qualitative research to propose candidates, set thresholds from logs (what counts as a bounce or a successful click), and write interpretable definitions. Validation: show that changing the surrogate changes the outcome, using dedicated experiments, holdback groups, historical experiment corpora with known outcomes, and external studies. For off-product outcomes, a 'data bridge' pilot measures the real outcome directly (by observation, not survey) in a random sample with a non-user baseline and models how the in-product behaviour predicts it.

Attributes

Loss type
Vestigial
Effort
Weeks

Steps

  1. Start from the slow key outcome and trace the behaviours that must precede it (for example credential purchase requires course completion, which requires module completion).
  2. Propose candidate early behaviours from qualitative research and by comparing retained with churned users.
  3. Set thresholds from logs (such as dwell time defining a bounce, or excluding quick-backs from successful clicks) and write interpretable, hard-to-game definitions.
  4. For off-product outcomes, run a data-bridge pilot: measure the real outcome directly in a random sample with a non-user baseline, link it to in-product behaviour and model the relationship.
  5. Validate causally: run experiments designed to move the surrogate, check a corpus of past experiments with known outcomes, and use holdback groups to confirm that moving the surrogate moves the key outcome.
  6. Adopt validated surrogates as experiment metrics and revalidate periodically as the product and users change.

Sources

  • Designing for Behavior Change (Wendel, 2014)
  • Designing with Data (King et al., 2017)
  • Lean Analytics (Croll & Yoskovitz, 2013)
  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
  • The Ultimate Question 2.0 (Reichheld & Markey, 2011)
  • Methods for Applying Predictive Modeling to Survey Data (Yih, 2024)
  • Building a Quant UXR Practice from Inception to Integration (Bakhshi, 2024)
  • Practical Statistics for Data Scientists (Bruce et al., 2020)

Also known as

  • Driver metric development and causal validation
  • Proxy metric derivation
  • Leading indicator derivation
  • Surrogate metric selection
  • Leading-indicator (aha moment) analysis
  • Activation metric discovery
  • Data bridge
  • Proxy-to-outcome calibration

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference