Change-Point Detection

Description

Statistically locates the points in a time series where its mean, variance or trend changes, distinguishing real shifts from noise.

Metric dashboards invite people to see changes in noise. Change-point methods formalise detection: CUSUM accumulates deviations from a target and signals when they exceed a threshold; offline methods such as PELT or binary segmentation find the optimal set of change locations with a penalty for complexity; Bayesian online methods give a running probability that a change has just occurred. Applied to UX metrics (task success, satisfaction, error rates, support contacts), they identify when behaviour shifted, which can then be linked to releases or external events, and they support alerting on real regressions.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Days

Steps

  1. Choose the metric series and its time resolution; consider seasonality and remove it if needed.
  2. Specify the type of change to detect (mean, variance, slope) and an appropriate cost function.
  3. Run offline detection (e.g. PELT with a BIC-like penalty) on historical data to find past changes.
  4. Check sensitivity of detected points to the penalty and validate against known events.
  5. For monitoring, configure CUSUM or Bayesian online detection with an acceptable false alarm rate.
  6. Investigate each detected change with release notes, segment breakdowns and qualitative data.

Sources

  • Page, Continuous inspection schemes (1954), Biometrika
  • Killick, Fearnhead & Eckley, Optimal detection of changepoints with a linear computational cost (2012), JASA
  • Adams & MacKay, Bayesian online changepoint detection (2007), arXiv
  • van der Maas, H., Complex Systems Research in Psychology (2024)

Also known as

  • Changepoint analysis
  • CUSUM
  • PELT
  • Bayesian online change-point detection

Categories

  • Analytics & Behaviour
  • Statistics & Modelling