Description
Estimates models from behavioural or survey data with missing values by iterating expectation and maximisation steps, avoiding the bias of deleting incomplete users.
Behavioural and survey data in UX are routinely incomplete: users drop out of diary studies, abandon multi-step surveys or sessions end early. Deleting incomplete cases (listwise deletion) or filling gaps with means biases results whenever missingness relates to the outcome, such as frustrated users quitting. The expectation-maximisation algorithm estimates parameters by alternating an E-step, computing expected values of the missing or latent quantities given current parameters, and an M-step, updating parameters to maximise likelihood given those expectations, until convergence. It provides maximum-likelihood estimates under the missing-at-random assumption and underpins mixture models and latent variable models. Good practice diagnoses missingness patterns, states the assumption, compares results with complete-case analysis and, where uncertainty must be propagated, uses multiple imputation alongside.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Extinct
- Goal
- Measure behaviour
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Frame incomplete session data as a latent variable problem: complete session outcome is the latent variable; observed partial session data is the observation.
- Apply EM: E-step estimates the probability of each possible complete session given the partial data; M-step updates model parameters.
- Compare: EM-estimated completion rates vs. naïve complete-case analysis.
- Large differences indicate non-random missingness (MNAR — missing not at random).
Sources
- Pattern Recognition and Machine Learning — Bishop
- Little & Rubin, Statistical Analysis with Missing Data
- Dempster, Laird & Rubin, Maximum likelihood from incomplete data via the EM algorithm (JRSS B, 1977)
Also known as
- EM algorithm
- Maximum likelihood with missing data
- EM imputation
Categories
- Analytics & Behaviour
- Statistics & Modelling