Multilevel (Hierarchical) Models

Description

Models data nested in groups (tasks within participants, users within companies, sessions within users) with random effects, giving correct uncertainty and partial pooling.

UX data are almost always nested: multiple tasks per participant, many sessions per user, users within accounts, ratings by several raters. Treating these as independent observations understates uncertainty and inflates false positives; averaging per participant throws away information. Multilevel models include random intercepts and slopes for grouping factors, estimate variance at each level (and the intraclass correlation), and partially pool estimates so small groups borrow strength from the whole. Crossed random effects for participants and items (tasks, stimuli) allow generalisation beyond the specific tasks tested, a point emphasised in psycholinguistics. Bayesian versions (brms, Stan) handle small samples well.

Attributes

Loss type
Latent
Effort
Days

Steps

  1. Identify the nesting and crossing structure: participants, tasks, sessions, accounts, raters.
  2. Specify the model with fixed effects for design conditions and random effects for grouping factors, including random slopes where justified.
  3. Choose an appropriate likelihood (e.g. log-normal for task times, binomial for success, ordinal for ratings).
  4. Fit with lme4, glmmTMB or brms; check convergence and simplify random structure if needed.
  5. Report fixed effects with intervals, variance components and ICCs.
  6. Use posterior or model-based predictions for new participants and tasks to show generalisable effects.

Sources

  • Raudenbush & Bryk, Hierarchical Linear Models (2nd ed., 2002)
  • Gelman & Hill, Data Analysis Using Regression and Multilevel/Hierarchical Models (2007)
  • Baayen, Davidson & Bates, Mixed-effects modeling with crossed random effects for subjects and items (2008), Journal of Memory and Language

Also known as

  • Mixed-effects models
  • Hierarchical linear models
  • HLM
  • Random effects models
  • Partial pooling

Categories

  • Analytics & Behaviour
  • Statistics & Modelling