Description
Hides the real chart among charts made from randomly re-paired data; if viewers can pick out the real one, the pattern is unlikely to be chance.
People readily see patterns in noise. The lineup protocol turns visual inspection into a test. The real scatterplot is placed among decoy plots in which the y values have been randomly shuffled against the x values, which destroys any association while keeping each variable's distribution. If the real plot can be singled out, the association is real; if it cannot, the pattern may be imagined. With 19 decoys and one independent viewer who did not see the data, a correct pick corresponds to a significance level of 1 in 20. Stine and Foster present it as the visual analogue of chi-square, which compares an observed table with one of no association. The same idea extends to any null model: residual plots, maps or time series simulated under the null. Once association is accepted, its direction, curvature, spread and outliers are described.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Evaluation, Post-launch
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Draw the chart of interest, such as the outcome against a predictor or a residual plot.
- Make decoy charts by shuffling y against x, or by simulating from the null model; use 19 decoys for a 5% test (the book's teaching version uses three).
- Arrange real and decoy charts in a grid, in random position and without labels.
- Ask one or more viewers who have not seen the real chart to pick the one that looks most different.
- If the real chart is picked, accept the pattern; with several viewers, compute the probability of that many correct picks by chance.
- If accepted, describe direction, curvature, variation and outliers.
Sources
- Statistics for Business: Decision Making and Analysis (Stine & Foster, 2018)
Also known as
- Visual test for association (lineup of scrambled scatterplots)
- Lineup test
- Lineup protocol
- Graphical inference with permuted plots
- Visual permutation test
Categories
- Analytics & Behaviour
- Statistics & Modelling