Reinforcement Learning Models of User Behaviour

Description

Fit a simple learning-from-reward model to people's sequences of choices to estimate how fast they learn from outcomes and how much they explore.

Reinforcement learning describes an agent that tries actions, receives rewards or penalties and gradually prefers what pays off. In psychology the same equations are fitted to people's trial-by-trial choices, producing interpretable parameters: a learning rate (how much one outcome shifts behaviour), an exploration or temperature parameter (how often people try something other than the best-known option) and sometimes separate rates for gains and losses. In UX this turns logs of repeated decisions into measures of learning rather than just conversion. You can ask whether a new feedback design makes people learn faster, whether some segments stop exploring early and miss features, or whether a recommendation system is teaching users to stop trying alternatives. It needs repeated choices with observable outcomes, which many products already log.

Attributes

Loss type
Siloed
Effort
Weeks

Steps

  1. Find a repeated choice with a visible outcome: which recommendation to open, which tool to use, which route to take.
  2. Extract per-user sequences of choices and outcomes from logs or a task study.
  3. Specify a simple model, such as Q-learning with a softmax choice rule, with a learning rate and an exploration parameter.
  4. Fit the parameters per user or per group by maximum likelihood or hierarchical Bayesian estimation.
  5. Check the model by simulating choices from the fitted parameters and comparing them with real behaviour.
  6. Compare parameters between designs or segments, and relate them to outcomes such as retention.

Sources

  • Cognitive Modeling: Nine Paradigms (slide deck)
  • Sutton, R. & Barto, A., Reinforcement Learning: An Introduction (2018)

Also known as

  • Computational reinforcement learning analysis of choices
  • Q-learning model fitting

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning