Description
At the end of a study, participants re-rate their starting state, so that pre and post ratings share the same internal standards and response shift can be detected.
Self-ratings assume people use the same yardstick before and after an intervention, but experience changes the yardstick. Response shift takes three forms: recalibration (internal standards move), reprioritisation (the importance of components changes) and reconceptualisation (the meaning of the construct changes). In the thentest, participants complete the usual posttest and then, immediately afterwards, a retrospective version rating how they were at the start. Change is estimated as posttest minus retrospective pretest, and comparing this with the conventional pre-post difference reveals recalibration. Ring and colleagues illustrate the stakes: a pain rating falling from 7 to 3 implies an effect of 4, but if the starting pain is later re-rated as 5 or 10 the effect becomes 2 or 7. For reprioritisation, participants also rate the relative importance of constructs before and after. Retrospective ratings depend on memory and on expectations of improvement.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Unmigrated
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Identify self-rated constructs whose meaning or standards are likely to shift during the study (competence, confidence, wellbeing, ease).
- Administer the conventional pretest at baseline.
- At the end, administer the normal posttest.
- Immediately after, administer a retrospective pretest asking participants to rate their state at the start of the study.
- Estimate change as posttest minus retrospective pretest and compare it with the conventional pre-post difference to detect recalibration.
- For reprioritisation, have participants rate the relative importance of each construct at both points.
Sources
- Advances in Longitudinal HCI Research (Karapanos et al., 2021)
Also known as
- Then-test
- Retrospective pretest-posttest design
- Response shift assessment
Categories
- Research Operations
- Research Design & Rigor