Fractional Factorial Experiments

Description

Tests several design factors simultaneously in a carefully chosen subset of combinations, estimating main effects and key interactions with far fewer runs.

A full factorial with k two-level factors needs 2^k combinations. Fractional factorials run a fraction (e.g. 2^(k-p)) chosen so that main effects and important interactions remain estimable, with known aliasing patterns described by resolution (Resolution IV: main effects clear of two-way interactions). This is vastly more efficient than one-factor-at-a-time testing, and it reveals interactions, such as a headline that only works with a particular image. Web 'multivariate testing' often runs full factorials without design, or runs many separate A/B tests and misses interactions. Proper DoE plans the fraction, randomises, analyses with ANOVA or regression and confirms surprising interactions.

Attributes

Loss type
Vestigial
Effort
Weeks

Steps

  1. Select factors (e.g. 5 design elements) and levels (usually 2 each) based on hypotheses.
  2. Choose a fractional design with adequate resolution (e.g. 2^(5-1) Resolution V, 16 cells) and document the alias structure.
  3. Randomise users to cells and compute sample size per cell for target effect sizes.
  4. Analyse with regression including main effects and estimable interactions; use half-normal plots to spot active effects.
  5. Interpret aliasing carefully and run a confirmatory follow-up if a key effect is aliased.
  6. Combine the best levels and validate the combined design against control.

Sources

  • Box, Hunter & Hunter, Statistics for Experimenters (2nd ed., 2005)
  • Fisher, The Design of Experiments (1935)
  • Montgomery, Design and Analysis of Experiments (9th ed., 2017)

Also known as

  • Factorial design
  • Design of experiments
  • DoE
  • Multivariate testing (proper)

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference