Thentest (Retrospective Pretest) for Response Shift

Description

At the end of a study, participants re-rate their starting state, so that pre and post ratings share the same internal standards and response shift can be detected.

Self-ratings assume people use the same yardstick before and after an intervention, but experience changes the yardstick. Response shift takes three forms: recalibration (internal standards move), reprioritisation (the importance of components changes) and reconceptualisation (the meaning of the construct changes). In the thentest, participants complete the usual posttest and then, immediately afterwards, a retrospective version rating how they were at the start. Change is estimated as posttest minus retrospective pretest, and comparing this with the conventional pre-post difference reveals recalibration. Ring and colleagues illustrate the stakes: a pain rating falling from 7 to 3 implies an effect of 4, but if the starting pain is later re-rated as 5 or 10 the effect becomes 2 or 7. For reprioritisation, participants also rate the relative importance of constructs before and after. Retrospective ratings depend on memory and on expectations of improvement.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. Identify self-rated constructs whose meaning or standards are likely to shift during the study (competence, confidence, wellbeing, ease).
  2. Administer the conventional pretest at baseline.
  3. At the end, administer the normal posttest.
  4. Immediately after, administer a retrospective pretest asking participants to rate their state at the start of the study.
  5. Estimate change as posttest minus retrospective pretest and compare it with the conventional pre-post difference to detect recalibration.
  6. For reprioritisation, have participants rate the relative importance of each construct at both points.

Sources

  • Advances in Longitudinal HCI Research (Karapanos et al., 2021)

Also known as

  • Then-test
  • Retrospective pretest-posttest design
  • Response shift assessment

Categories

  • Research Operations
  • Research Design & Rigor