Matrix completion (Hard-Impute)

Description

Fills missing entries of a data matrix by repeatedly fitting a low-rank principal-components approximation and replacing gaps with its predictions until the fit stops improving.

Hard-Impute treats a data matrix with gaps as approximately low rank. Each column is standardised and missing entries start at their column means; the first M principal components of the completed matrix are computed, missing entries are replaced by their rank-M approximation, and the squared error on observed entries is recalculated. The loop repeats until that error stops falling. The same machinery serves two purposes: imputing scattered missing values before analysis, and recommendation, where most entries are missing by design (the Netflix matrix was about 99% empty) and user scores and item loadings read as memberships in 'cliques' and 'genres'. The number of components is chosen by hiding some observed entries and checking how well they are recovered. The method assumes values are missing at random and finds a local, not global, optimum.

Attributes

Loss type
Siloed
Effort
Days

Steps

  1. Standardise each column and fill every missing value with its column mean of observed values.
  2. Compute the first M principal components of the completed matrix (its rank-M approximation).
  3. Replace each originally missing entry with its rank-M approximation.
  4. Recompute squared error on observed entries only and repeat the previous two steps until it stops decreasing.
  5. Choose M by masking a further 5-10% of observed entries, re-running, and picking the rank that recovers them best.
  6. Return the imputed matrix with approximate component scores and loadings, and check that missingness is plausibly random before using them.

Sources

  • An Introduction to Statistical Learning with Applications in Python (James et al., 2023)
  • A User's Guide to Principal Components (Jackson, 1991)

Also known as

  • Hard-Impute
  • Matrix completion by iterative principal components
  • Low-rank matrix completion
  • PCA imputation
  • Matrix factorisation for recommendations

Categories

  • Analytics & Behaviour
  • Statistics & Modelling