Triggered analysis

Description

Analyse only the users who could have been affected by an experiment's change, gaining power, then dilute the effect back to the whole population for decisions.

Many experiments change something only a fraction of users ever encounter, such as a checkout step or an edge case. Including everyone adds noise from unaffected users and hides real effects. Triggered analysis restricts the analysis to units who reached a point where the variants could differ, identified by targeted segment, reaching the changed step, coverage differences, or counterfactual logging of what the other variant would have shown. Trigger definitions must rely on pre-experiment state, and triggered users stay in the analysis from first trigger onward. The gain can be large: triggering on checkout starters roughly halved the users needed (from 121,600 to 64,000). For decisions, the triggered absolute delta is diluted: multiplied by the triggered count and divided by the overall total. Validity checks include an SRM test on triggered users and an A/A-like comparison of never-triggered users.

Attributes

Loss type
Latent
Effort
Days

Steps

  1. Define the trigger as any point where the variants could produce different experiences, using pre-experiment state only.
  2. Log triggers at runtime in both variants, including counterfactual outputs where the change depends on computed results.
  3. Analyse triggered users from their first trigger onward, keeping them in for the rest of the run.
  4. Run a sample ratio mismatch check on triggered users and an A/A-like analysis of never-triggered users, which should show no difference.
  5. Compute the diluted overall impact: triggered absolute delta multiplied by triggered count, divided by the overall total.
  6. If counterfactual logging is costly, run an A/A'/B design to measure its performance cost.

Sources

  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)

Also known as

  • Triggering
  • Counterfactual triggering
  • Diluted impact

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference