Metric-wellbeing gap measurement

Description

Quantifies the divergence between the metric a product optimises and users' measured wellbeing or long-term outcomes, showing when success on the metric masks harm.

Products often succeed on engagement metrics while users feel worse or make poorer decisions. This method pairs behavioural metrics with validated wellbeing or outcome measures for the same users (such as anxiety, decision confidence, financial health or regret) collected through surveys or experience sampling, and models the relationship. A gap exists when high metric values coincide with flat or worse wellbeing, or when increments in the metric yield diminishing or negative wellbeing returns. Rigour requires validated wellbeing instruments, linking at the user level with consent, longitudinal measurement to separate cause from selection, and segment-level analysis because harms often concentrate in vulnerable groups.

Attributes

Loss type
Latent
Data
Mixed
Effort
Months

Steps

  1. Select the optimised metric (e.g. weekly sessions) and validated wellbeing or outcome measures relevant to the domain (e.g. a financial anxiety scale).
  2. Recruit a consented panel of at least 300-500 users and link survey responses to behavioural data.
  3. Measure wellbeing at baseline and again after 4-12 weeks, or use experience sampling around product use.
  4. Model wellbeing as a function of the metric, plotting the relationship and testing for non-linearity.
  5. Compute a gap index, such as the proportion of high-metric users in the bottom wellbeing quartile, by segment.
  6. Report the gap with guardrail recommendations and, where possible, test alternative designs experimentally.

Sources

  • The Alignment Problem (Brian Christian)
  • Artificial Intelligence for Humans (Jeff Heaton)
  • Methods Atlas (Vivek Gopinath)
  • Diener (1984) Subjective well-being
  • Peters, Calvo & Ryan (2018) Designing for motivation, engagement and wellbeing in digital experience
  • The Alignment Problem (Christian)
  • AI for Humans (Heaton)
  • Methods Atlas (Gopinath)
  • The Inquiry Series, Ch 58 — AI Explainability & Value Alignment Audit

Also known as

  • Proxy-outcome gap analysis
  • Engagement-wellbeing divergence

Categories

  • Analytics & Behaviour
  • Product Analytics & Strategy