Regression diagnostics

Description

A systematic check of a fitted regression for violated assumptions, influential points and collinearity so its coefficients and predictions can be trusted.

Regression diagnostics test whether a model is fit for the claims made from it. They examine residuals for non-linearity and unequal variance, check normality of residuals for small-sample inference, find influential observations through leverage and Cook's distance, and measure multicollinearity through variance inflation factors. In UX data, common culprits are skewed task times, ceiling effects on satisfaction scales, a few expert participants dominating the fit and repeated measures treated as independent. Rigour means running diagnostics routinely, documenting remedies (log-transforming time, robust standard errors, mixed models) and reporting whether conclusions change after them, rather than silently deleting awkward points.

Attributes

Loss type
Vestigial
Effort
Hours

Steps

  1. Plot residuals against fitted values to check linearity and constant variance; a funnel shape signals heteroscedasticity.
  2. Inspect a Q-Q plot of residuals and, for task time, consider log transformation before modelling.
  3. Compute leverage and Cook's distance and investigate any point above 4/n or with a Cook's D above 1.
  4. Compute VIFs and treat values above 5 as a warning and above 10 as serious collinearity.
  5. Check independence: if participants contribute multiple observations, refit with clustered errors or a mixed model.
  6. Refit with remedies (transformations, robust or HC3 standard errors, removing justified outliers) and report whether conclusions change.

Sources

  • Applied Predictive Modeling (Max Kuhn & Kjell Johnson)
  • R for Marketing Research and Analytics (Chris Chapman & Elea McDonnell Feit)
  • Regression Diagnostics (Belsley, Kuh & Welsch)
  • Applied Predictive Modeling (Kuhn & Johnson)
  • R for Marketing Research (Chapman & Feit)
  • The Inquiry Series, Ch 43 — Regression, Moderation & Mediation for UX Researchers
  • Practical Statistics for Data Scientists (Bruce & Bruce)

Also known as

  • Model diagnostics
  • Residual analysis
  • Assumption checking
  • Cook's distance
  • Partial residual plots
  • Regression Diagnostics (Outliers, Influence, Heteroskedasticity)

Categories

  • Analytics & Behaviour
  • Statistics & Modelling