Guardrail metric monitoring

Description

Tracks metrics that must not deteriorate while a primary metric is optimised, with thresholds that trigger investigation or rollback.

Guardrail metrics protect against shipping changes that win on the target metric but harm something important: latency, error rates, support contacts, unsubscribes, accessibility, user wellbeing or revenue. In experimentation, they are pre-specified with non-inferiority margins; after launch, they are monitored continuously with alerting thresholds. They echo safety monitoring in clinical trials, where efficacy is never judged without harm data. Rigour requires choosing guardrails that capture plausible harms identified through research, setting margins before data arrives, powering tests to detect meaningful degradation, and making escalation paths explicit. UX research contributes by identifying user-experience harms that standard engineering guardrails miss.

Attributes

Loss type
Active
Stage
Post-launch
Effort
Days

Steps

  1. Identify plausible harms of the change from research, risk review and past incidents.
  2. Select 3-6 guardrail metrics covering system health, user experience and trust or wellbeing.
  3. Set non-inferiority margins or alert thresholds before launch (e.g. support contacts may not rise by more than 5% relative).
  4. Power experiments so a breach of the margin can be detected, using one-sided tests for degradation.
  5. Monitor guardrails in dashboards with automated alerts and a named owner.
  6. On a breach, pause or roll back and investigate with qualitative follow-up before resuming.

Sources

  • Storytelling with Data (Cole Nussbaumer Knaflic)
  • Articulating Design Decisions (Tom Greever)
  • Envisioning Information (Edward Tufte)
  • Trustworthy Online Controlled Experiments (Ron Kohavi, Diane Tang & Ya Xu)
  • Storytelling with Data (Knaflic)
  • Articulating Design Decisions (Greever)
  • Envisioning Information (Tufte)
  • The Inquiry Series, Ch 59 — Research Communication — From Evidence to Strategic Brief

Also known as

  • Guardrail metrics
  • Counter-metrics
  • Do-no-harm metrics

Categories

  • Analytics & Behaviour
  • Product Analytics & Strategy