Mixed-Method Counterfactual Framework for Complex Interventions

Description

A two-step way to estimate what would have happened without a complex, multi-actor intervention: choose attribution, contribution or substitution analysis, then combine several counterfactual approaches.

Complex interventions have many components and funders, change in nonlinear ways and shift their outcomes, so experimental rigour is rarely possible. Bamberger's framework first sets the level of claim the evidence can support. Attribution analysis needs a matched comparison group and credits differences to the intervention. Contribution analysis asks what one actor added to a change produced jointly with others. Substitution analysis estimates the net extra resources an input created after diversion or multiplier effects. The second step defines the counterfactual by combining approaches: statistical comparison, theory-based comparison of predicted and observed processes with rival hypotheses tested, participatory judgement such as Most Significant Change, and expert or stakeholder rating scales. The design is strengthened by splitting the programme into evaluable parts, portfolio analysis, reconstructed baselines and secondary data, and estimates are triangulated.

Attributes

Loss type
Unmigrated
Data
Mixed
Effort
Months

Steps

  1. Assess what comparison group, if any, is available.
  2. Choose attribution analysis (matched comparison exists), contribution analysis (several actors produce the outcome jointly) or substitution analysis (the question is net added resources).
  3. Select one or more counterfactual approaches: statistical, theory-based (theory of change versus observed, testing rival explanations), participatory (for example Most Significant Change) and rating scales.
  4. Strengthen the design by breaking the intervention into evaluable components, analysing the portfolio, reconstructing baselines and drawing on secondary data and other studies.
  5. Triangulate the estimates and report the effect at the chosen level with the evidence behind it.

Sources

  • Introduction to Mixed Methods in Impact Evaluation (Bamberger, 2012)

Also known as

  • Attribution, contribution and substitution analysis
  • Defining the counterfactual for complex interventions

Categories

  • Analytics & Behaviour
  • Experimentation & Causal Inference