Data Distribution Shift Detection and ML Monitoring

Description

Compare production inputs and predictions with a reference window using schema checks, summary statistics and two-sample tests, to catch data shifts that silently degrade a model-driven feature.

Models degrade when production data drift from training data. Shifts may be covariate (inputs change), label (outcome rates change) or concept (the input-outcome relation changes). Where labels or user feedback arrive, accuracy can be monitored directly; otherwise one monitors predictions, features and raw inputs. Feature validation checks values against expected schemas (ranges, formats, allowed sets). Summary statistics are a start, but two-sample tests between a reference and a current window are better: Kolmogorov-Smirnov for one-dimensional quantities such as prediction scores, and MMD or LSDD for multivariate features after dimensionality reduction. Window choice matters: cycles shorter than the window are missed, short windows raise false alarms, and cumulative metrics hide dips that sliding metrics reveal. Statistical significance is not practical importance, and KS in particular raises many false positives on large samples.

Attributes

Loss type
Siloed
Stage
Post-launch
Effort
Weeks

Steps

  1. Log user feedback, predictions, input features and raw inputs with timestamps.
  2. Validate features against expected schemas: ranges, formats, allowed values and relations between fields.
  3. Compare summary statistics of the current window with a reference window.
  4. Run two-sample tests on predictions (for example Kolmogorov-Smirnov) and on dimension-reduced features (MMD or LSDD).
  5. Choose windows that respect daily and weekly cycles, use sliding rather than cumulative statistics, and merge short windows for longer views.
  6. Alert on shifts that are both significant and practically large, then decide whether to retrain, adapt or investigate the product change behind them.

Sources

  • Designing Machine Learning Systems (Huyen, 2022)

Also known as

  • Drift detection
  • Covariate shift detection
  • Two-sample drift tests
  • Feature validation

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning