Computerised Adaptive Testing (CAT)

Description

Selects each next question from a calibrated item bank to be most informative at the respondent's current estimated level, stopping at a fixed length or target precision.

A fixed questionnaire wastes items that are far from a respondent's level: people with very high satisfaction learn little from items that only discriminate at the low end. A computerised adaptive test needs five components: a calibrated IRT item bank, a starting rule, an item selection rule, a stopping rule and a scoring method. Item information is additive and roughly the inverse of the squared standard error, so precision at a level depends on how many items target it. Maximum-information selection picks the item most informative at the provisional estimate; before responses vary, MAP or EAP estimation with a prior keeps that estimate finite. Fixed-length stopping gives uniform length; precision-based stopping gives shorter questionnaires for most people and suits cut-score decisions, but is limited by bank coverage. Health outcome systems such as PROMIS use CAT to measure constructs precisely with four to eight items.

Attributes

Loss type
Unmigrated
Effort
Months

Steps

  1. Build and calibrate an item bank with an IRT model on a large, representative sample (typically several hundred to over a thousand respondents), checking unidimensionality and fit.
  2. Choose a starting item or starting estimate (for example the population mean).
  3. After each response, update the provisional trait estimate with EAP or MAP.
  4. Select the next item with maximum information at that estimate, subject to any content-balance or exposure constraints.
  5. Stop at a fixed number of items or when the standard error falls below a target (for example 0.3 on the theta scale).
  6. Report the final IRT score and standard error, and monitor item exposure and bank drift over time.

Sources

  • The SAGE Handbook of Quantitative Methods in Psychology (Millsap & Maydeu-Olivares, 2009)

Also known as

  • Tailored testing
  • Adaptive questionnaire
  • IRT-based adaptive survey
  • Computerized adaptive testing

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics