Trust in Automation Scale (Jian et al.)

Description

A 12-item empirically derived scale measuring trust and distrust in automated systems, used with reliability data to assess whether trust is calibrated.

Jian, Bisantz and Drury derived their scale empirically from word-elicitation and cluster studies, yielding 12 items on a 7-point scale covering deception, suspicion, harmful outcomes, confidence, integrity, dependability and familiarity. Lee and See's theory adds that trust should be calibrated: matched to the actual reliability of the automation. Rigorous use therefore pairs the scale with objective measures of system reliability and of reliance behaviour (acceptance of correct and incorrect recommendations). Over-trust appears as accepting wrong advice; under-trust as rejecting correct advice. This turns 'do users trust our AI?' into 'is their trust appropriate?', which is the question that matters.

Attributes

Loss type
Latent
Effort
Weeks

Steps

  1. Define the automation's actual reliability on the task set, including a planned proportion of errors (e.g. 10-30%).
  2. Administer the 12-item scale before and after exposure, and at intervals in longitudinal studies.
  3. Log reliance behaviour: agreement with correct suggestions, agreement with incorrect suggestions, verification actions.
  4. Compute calibration: compare trust scores and reliance rates with actual reliability; plot trust trajectories after errors.
  5. Test design interventions such as confidence displays or explanations for their effect on calibration, not just trust level.
  6. Report over-reliance and under-reliance rates alongside the scale score.

Sources

  • Jian, Bisantz & Drury, Foundations for an empirically determined scale of trust in automated systems (2000), International Journal of Cognitive Ergonomics
  • Lee & See, Trust in automation: Designing for appropriate reliance (2004), Human Factors

Also known as

  • Checklist for Trust between People and Automation
  • Jian trust scale
  • Calibrated trust measurement

Categories

  • Accessibility
  • Human Factors & Error