Strategy of Elaboration for Ruling Out Threats to Validity

Description

When a named threat could bias an effect estimate, adds a second, contrasting estimate whose mix of effect and threat differs, and compares the two to separate them.

Any effect estimate equals the treatment effect plus the effect of whatever threats are present. Reichardt's elaboration strategy designs a second estimate to pull them apart. In estimate-and-subtract, the second estimate contains only the threat (for example a comparison series affected by the same history but not the treatment), so subtracting it removes the threat. In vary-the-size-of-the-treatment-effect, the second estimate contains the same threat but a different amount of effect; the effect is demonstrated to the extent the estimates differ. In vary-the-size-of-the-threat, the second contains the same effect but a different, ideally opposite, amount of threat; the threat is ruled out to the extent the estimates agree. The second estimate can come from different recipients, times, settings or outcome measures, the same four factors that underpin Reichardt's principle of parallelism. This is distinct from Lazarsfeld's elaboration model, which introduces control variables into cross-tabulations to interpret an association.

Attributes

Loss type
Latent
Data
Mixed
Effort
Days

Steps

  1. Name the specific threat (history, seasonality, selection, instrumentation) that could bias the first estimate.
  2. Choose the source of a second estimate: different users, a different time, a different setting or market, or a different outcome measure.
  3. Arrange that the second estimate contains only the threat, a different amount of treatment effect, or a different and ideally opposite amount of the threat.
  4. Compute both estimates using the same procedure.
  5. Subtract (estimate-and-subtract) or compare the sizes of the estimates to separate the treatment effect from the threat.
  6. Repeat for each other plausible threat and report which were ruled out, bounded or left open.

Sources

  • The SAGE Handbook of Quantitative Methods in Psychology (Millsap & Maydeu-Olivares, 2009)

Also known as

  • Reichardt's elaboration strategy
  • Estimate-and-subtract
  • Vary-the-size-of-the-treatment-effect
  • Vary-the-size-of-the-threat
  • Principle of parallelism

Categories

  • Research Operations
  • Research Design & Rigor