Tactics for Testing or Confirming Findings

Description

A named set of 13 checks for qualitative conclusions, covering data quality, exceptions to patterns, and sceptical tests of explanations, framed against known judgement biases.

Miles and Huberman start from evidence that human judgement is biased: people find patterns in random data, overweight vivid instances and seek confirming cases. Field researchers face three archetypal biases: the holistic fallacy (seeing events as more patterned than they are), elite bias (overweighting articulate, high-status informants) and going native. They then set out 13 tactics. Data quality: (1) checking for representativeness, for example by adding cases, sampling contrasting cases and sampling randomly within the universe; (2) checking for researcher effects, both the researcher's effect on the site and the site's effect on the researcher; (3) triangulating; (4) weighting the evidence. Looking at 'unpatterns': (5) checking the meaning of outliers; (6) using extreme cases; (7) following up surprises; (8) looking for negative evidence. Testing explanations: (9) making if-then tests, stating conditional predictions and looking to see whether the 'then' holds; (10) ruling out spurious relations, where a third variable causes both A and B; (11) replicating a finding; (12) checking out rival explanations. Finally, (13) getting feedback from informants. For UX teams, whose insights often rest on a few sessions, the list works as a review checklist before findings go to a roadmap.

Attributes

Loss type
Latent
Effort
Days

Steps

  1. Before finalising, list each key finding and the evidence behind it.
  2. Check data quality: is the sample representative, could the researcher or session setup have produced it, does it triangulate, how strong is each source?
  3. Examine outliers, extreme cases, surprises and negative evidence rather than setting them aside.
  4. State if-then predictions from each explanation and check them against data not yet used.
  5. Look for third variables that could explain an apparent relationship, and try replicating the finding in another segment or study.
  6. Write down rival explanations and test them, then take the findings back to participants for feedback.

Sources

  • Miles, M. & Huberman, A. M., Qualitative Data Analysis: An Expanded Sourcebook, 2nd ed. (1994)

Also known as

  • Thirteen tactics for confirming findings
  • Miles and Huberman verification tactics
  • Qualitative verification tactics

Categories

  • Research Operations
  • Research Design & Rigor