Index-versus-Fundamental Measurement Check for Proxy Measures

Description

Before treating a physical measure such as task time as an interval or ratio measure of a psychological attribute such as effort, check whether any empirical law links the two.

UX research routinely uses physical quantities as stand-ins for psychological ones: seconds for effort, clicks for difficulty, dwell time for interest. Time is a ratio scale because intervals add, but nothing about effort corresponds to adding seconds, so the log or square of time indexes effort just as well. Without a law that ties the proxy's structure to the attribute, as Hooke's law ties a spring's stretch to weight, the proxy is an index carrying only ordinal information about the attribute. Conclusions such as 'twice as much effort' or 'the effect is larger for novices' then depend on an arbitrary choice of transformation. Luce and colleagues recommend checking which conclusions survive monotone rescaling, and note that collecting the index across two crossed factors and testing for an additive conjoint structure is one route to a genuine interval measure.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Name the psychological attribute of interest and the physical index used for it (time, counts, force, distance).
  2. Ask whether a theory or empirical law links the index's own structure to the attribute; if none exists, treat the index as ordinal for that attribute.
  3. List the planned conclusions and recompute each under monotone transforms such as log and square; keep only those whose direction survives.
  4. Prefer medians, rank-based tests and ordinal effect sizes for claims about the attribute, and avoid claims about ratios or differences of the attribute.
  5. Check that alternative indices of the same attribute (time, errors, self-rated effort) are monotonically related in the conditions studied.
  6. To go beyond ordinal, collect the index across two crossed manipulated factors and test whether an additive conjoint representation fits.

Sources

  • Foundations of Measurement, Vol. 3: Representation, Axiomatization, and Invariance (Luce et al., 1990)

Also known as

  • Proxy index scale-type check
  • Pointer versus index measurement check

Categories

  • Research Operations
  • Research Design & Rigor