Experiment test-cell design strategy

Description

Treat the set of hypotheses and test cells as a design task: write structured hypotheses, choose cells that each teach something distinct, test extremes first, and plan follow-ups.

Most A/B tests compare one shippable variant with control, which teaches little when it fails. This strategy designs experiments to learn. Hypotheses follow a template ('For [user group], if [change] then [effect] because [rationale], which will impact [measure]') with a learning statement for success and failure, are generated broadly by varying their parts, and are pruned with estimates, past data, cheap research and feasibility checks (minimum detectable effect, sample, business cycle). Cells are chosen to span the dimensions of interest and tabulated against them; redundant cells are dropped. In exploration, a deliberately extreme, unshippable variant shows whether an idea can move the metric at all (Skyscanner dropped sharing when even a pop-up did nothing). For large redesigns, a global phase tests divergent concepts on a small traffic share until one nears control, then a local phase optimises it (Etsy gained about 5% conversion).

Attributes

Loss type
Latent
Effort
Weeks

Steps

  1. Write each hypothesis as 'for [user group], if [change] then [effect] because [rationale], which will impact [measure]', and add a learning statement for both success and failure.
  2. Generate broadly by holding some parts fixed and varying others (behaviours, changes, user groups), then prune with ballpark value estimates, past tests, disconfirming evidence and cheap research such as pilots or usability tests.
  3. Check feasibility for each survivor: minimum detectable effect, sample, allocation, behaviour lag and business cycle.
  4. Choose cells so they span the dimensions you want to learn about, tabulate each cell against those dimensions, and drop cells that teach nothing another cell does not.
  5. In exploration, include a realistic and a deliberately exaggerated variant; if even the extreme variant has no effect, drop the idea; for continuous variables test the extremes before finer steps.
  6. For large redesigns, run divergent concepts on a small share of traffic (Etsy used 5%) in two-week iterations with qualitative research until one comes close to control, then optimise it with smaller A/B tests until it wins.
  7. For each likely outcome, decide which experiment comes next and start designing follow-ups while results come in; keep unpursued and failed hypotheses on record.

Sources

  • Designing with Data (King et al., 2017)
  • Product Management for UX People (Crumlish, 2022)

Also known as

  • Test-cell portfolio design with a learning plan
  • Designing to learn
  • Cell-by-dimension table
  • Exploratory extreme-variant testing
  • Maximum-dose test
  • Testing extremes first
  • Two-phase redesign testing
  • Global exploration then local optimisation
  • Experiment hypothesis generation and prioritisation
  • Hypothesis statement
  • Learning statement

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference