Description
Tests several design factors simultaneously in a carefully chosen subset of combinations, estimating main effects and key interactions with far fewer runs.
A full factorial with k two-level factors needs 2^k combinations. Fractional factorials run a fraction (e.g. 2^(k-p)) chosen so that main effects and important interactions remain estimable, with known aliasing patterns described by resolution (Resolution IV: main effects clear of two-way interactions). This is vastly more efficient than one-factor-at-a-time testing, and it reveals interactions, such as a headline that only works with a particular image. Web 'multivariate testing' often runs full factorials without design, or runs many separate A/B tests and misses interactions. Proper DoE plans the fraction, randomises, analyses with ANOVA or regression and confirms surprising interactions.
Attributes
- Discipline
- Experimentation & Causal Inference
- Loss type
- Vestigial
- Stage
- Design, Evaluation
- Data
- Quantitative
- Effort
- Weeks
Steps
- Select factors (e.g. 5 design elements) and levels (usually 2 each) based on hypotheses.
- Choose a fractional design with adequate resolution (e.g. 2^(5-1) Resolution V, 16 cells) and document the alias structure.
- Randomise users to cells and compute sample size per cell for target effect sizes.
- Analyse with regression including main effects and estimable interactions; use half-normal plots to spot active effects.
- Interpret aliasing carefully and run a confirmatory follow-up if a key effect is aliased.
- Combine the best levels and validate the combined design against control.
Sources
- Box, Hunter & Hunter, Statistics for Experimenters (2nd ed., 2005)
- Fisher, The Design of Experiments (1935)
- Montgomery, Design and Analysis of Experiments (9th ed., 2017)
Also known as
- Factorial design
- Design of experiments
- DoE
- Multivariate testing (proper)
Categories
- Analytics & Behaviour
- Experimentation & Causal Inference