Experiment review meeting and design checklist

Description

A required pre-launch design checklist plus a standing cross-team meeting that vets upcoming experiments and reviews results first for trust, then for decisions.

Experimentation quality depends on governance as much as statistics. Before launch, each experiment completes a short checklist (hypothesis, key metric, practically meaningful change, power calculation) reviewed by an expert; Google required this until teams matured. A standing review meeting, separate from design critique and mixing product managers and designers who do not usually work together, covers two kinds of work. For upcoming experiments it questions the hypothesis and its supporting data, the methodology and its alternatives, how well the design expresses the hypothesis, and whether success metrics really test it. For launched experiments it first checks trustworthiness (sample ratio mismatch, guardrails, instrumentation), then results, transferable learning, anomalies and confounds, and records trade-offs in the overall evaluation criterion and institutional memory for meta-analysis.

Attributes

Loss type
Renamed
Effort
Hours

Steps

  1. Require each experiment to complete a short checklist before start: hypothesis, key metric, minimum meaningful change, and power calculation.
  2. Have an experienced reviewer sign off the checklist before launch.
  3. Hold a standing review meeting, separate from design critique, inviting product managers, designers and data specialists from different teams.
  4. For upcoming work, question the hypothesis, the data behind it, the method and its alternatives, the design and the success metrics.
  5. For launched work, review trustworthiness first (sample ratio mismatch, guardrail metrics, instrumentation), then results and launch recommendation.
  6. Discuss anomalies, possible confounds and what else could have been tested, and codify metric trade-offs into the overall evaluation criterion.
  7. Record decisions and learning in a searchable archive for institutional memory and meta-analysis; relax the checklist as maturity grows.

Sources

  • Designing with Data (King et al., 2017)
  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)

Also known as

  • Recurring experiment review meeting
  • Project review meeting
  • Experiment review
  • Experiment design checklist

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference