Process and Implementation Evaluation

Description

Study how a programme or rollout actually operates and what people experience, to check what was really delivered before judging its outcomes.

Process evaluation describes how a programme produces results from the perspectives of participants, staff and outsiders. Implementation evaluation asks what was actually delivered as opposed to what was planned. Patton criticises pure pre-post outcome studies as an 'empty box' that cannot say what produced, or failed to produce, results. Core questions are what participants experience, what services are given, what staff do, what it is like to be in the programme and how it is organised. The RAND Change Agent Study found 'mutual adaptation' between innovations and settings, and in one outreach programme the door-to-door work differed from its design. Process findings make null outcomes interpretable and isolate the critical elements for replication. In UX it is the counterpart to measuring launch metrics.

Attributes

Loss type
Siloed
Effort
Weeks

Steps

  1. Describe the programme or rollout as designed from documents, logic models and staff accounts.
  2. Observe activities and interview participants and staff about what actually happens.
  3. Ask the implementation questions: what users experience, what is delivered, what staff do and how it is organised.
  4. Compare observed practice with the design and record adaptations and their reasons.
  5. Report processes so that outcome findings can be linked to what was actually delivered, and identify critical elements.

Sources

  • Qualitative Research & Evaluation Methods (Patton, 2002)
  • User-Friendly Handbook for Mixed Method Evaluations (Frechtling & Sharp, 1997)
  • The Sage Handbook of Mixed Methods Research Design (Poth, 2023)
  • The SAGE Handbook of Qualitative Research (Denzin & Lincoln, 2017)
  • Quantitative Social Research Methods (Singh, 2007)
  • Introduction to Mixed Methods in Impact Evaluation (Bamberger, 2012)

Also known as

  • Process study
  • Implementation evaluation
  • Empty box critique
  • Mutual adaptation

Categories

  • Research Operations
  • Research Design & Rigor