Making and Testing Predictions (Two-Envelope Informant Feedback)

Description

Validate a qualitative case analysis by writing explicit predictions with supporting and spoiling factors, then months later asking informants what happened before showing them the prediction.

A prospective test of a qualitative explanation. From the causal network and matrices, the analyst writes one precise prediction per main outcome (usually four or five) with up to five weighted supporting factors and five possible spoilers. After 6 to 12 months the prediction is sent to at least two informants in different roles in two envelopes. Envelope one asks them to describe the current situation and rank its causes without seeing the prediction; envelope two then shows the prediction and asks them to rate each factor's pertinence. Comparing predicted and actual outcomes, and the overlap of reasons, tests the analysis while keeping informants blind to the analyst's reasoning. Having a second analyst generate predictions from the case report also tests how explicit it is. In the original school study 58% of assigned causes fully agreed with informants. Allow one to two days per complex case.

Attributes

Loss type
Latent
Effort
Months

Steps

  1. For each main outcome, write one precise prediction derived from the causal streams and related matrices.
  2. List and weight up to about five supporting factors and five possible spoilers.
  3. Review any possible harm to informants from the predictions.
  4. After 6-12 months send envelope one to at least two informants in different roles: describe the current situation and rank the contributing factors.
  5. Then send envelope two: the prediction with a pertinence rating table for each factor.
  6. Compare predicted and actual outcomes and the overlap of reasons, and follow up discrepancies by phone.
  7. Optionally have a second analyst make predictions from the case report to test its explicitness.

Sources

  • Miles, M. & Huberman, A. M., Qualitative Data Analysis: An Expanded Sourcebook, 2nd ed. (1994)

Also known as

  • Prediction testing of a case analysis
  • Two-envelope prediction check

Categories

  • Research Operations
  • Research Design & Rigor