Visual lineup test for association

Description

Hides the real chart among charts made from randomly re-paired data; if viewers can pick out the real one, the pattern is unlikely to be chance.

People readily see patterns in noise. The lineup protocol turns visual inspection into a test. The real scatterplot is placed among decoy plots in which the y values have been randomly shuffled against the x values, which destroys any association while keeping each variable's distribution. If the real plot can be singled out, the association is real; if it cannot, the pattern may be imagined. With 19 decoys and one independent viewer who did not see the data, a correct pick corresponds to a significance level of 1 in 20. Stine and Foster present it as the visual analogue of chi-square, which compares an observed table with one of no association. The same idea extends to any null model: residual plots, maps or time series simulated under the null. Once association is accepted, its direction, curvature, spread and outliers are described.

Attributes

Loss type
Siloed
Effort
Hours

Steps

  1. Draw the chart of interest, such as the outcome against a predictor or a residual plot.
  2. Make decoy charts by shuffling y against x, or by simulating from the null model; use 19 decoys for a 5% test (the book's teaching version uses three).
  3. Arrange real and decoy charts in a grid, in random position and without labels.
  4. Ask one or more viewers who have not seen the real chart to pick the one that looks most different.
  5. If the real chart is picked, accept the pattern; with several viewers, compute the probability of that many correct picks by chance.
  6. If accepted, describe direction, curvature, variation and outliers.

Sources

  • Statistics for Business: Decision Making and Analysis (Stine & Foster, 2018)

Also known as

  • Visual test for association (lineup of scrambled scatterplots)
  • Lineup test
  • Lineup protocol
  • Graphical inference with permuted plots
  • Visual permutation test

Categories

  • Analytics & Behaviour
  • Statistics & Modelling