ROC curve analysis for UI alerts

Description

Plots hit rate against false-alarm rate across alert thresholds or user confidence levels to choose the threshold that best balances misses against alarm fatigue.

An ROC curve shows the trade-off between true-positive and false-positive rates as a decision threshold varies. For UI alerts it can be applied at two levels: the system level (what detector threshold triggers an alert) and the human level (how users respond, using confidence ratings to trace their curve). The area under the curve (AUC) summarises discriminability independent of threshold; the optimal operating point depends on base rates and the costs of misses versus false alarms, via the slope beta = [P(noise)/P(signal)] x [(cost FA)/(value hit)]. The method makes explicit that every alert threshold is a value judgement and that high false-alarm rates degrade human trust and response, a key cause of alarm fatigue. Rigour comes from labelled ground truth and explicit cost weighting.

Attributes

Loss type
Siloed
Effort
Weeks

Steps

  1. Assemble a labelled dataset of events (true conditions versus benign) with the detector score for each, ideally at least 100 positives and 100 negatives.
  2. Sweep thresholds and compute true- and false-positive rates at each to plot the system ROC and AUC.
  3. Elicit stakeholder costs for misses and false alarms and the base rate, and compute the optimal operating slope.
  4. Run a user study where participants respond to alerts and give confidence ratings on a 6-point scale; construct the human ROC from cumulative rating thresholds.
  5. Compare the joint human-plus-system performance at candidate thresholds, noting positive predictive value; alerts with PPV below about 10-20% are associated with override rates above 90% in clinical settings.
  6. Select the threshold, document the trade-off, and set monitoring to recompute ROC after launch as base rates drift.

Sources

  • Van Drongelen, W., Signal Processing for Neuroscientists
  • MacKay, D. J. C., Information Theory, Inference, and Learning Algorithms (2003)
  • Gopinath, V., Forward-Thinking Playbook
  • Van Drongelen (Signal Processing for Neuroscientists)
  • MacKay
  • Forward-Thinking Playbook (Gopinath)
  • The Inquiry Series, Ch 2 — Signal, Noise & the Detection of Intent

Also known as

  • Receiver operating characteristic analysis
  • Alert threshold ROC
  • Alarm ROC analysis

Categories

  • User Research
  • Cognitive Psychology & Perception