Number Needed to Treat and Harm

Description

Expresses an intervention's effect as how many users must be exposed for one to benefit (NNT) or be harmed (NNH), making trade-offs concrete.

Relative effects (a 20% lift) are easily misunderstood. NNT is the reciprocal of the absolute risk reduction: if a design reduces abandonment from 10% to 8%, 50 users must see it to prevent one abandonment. NNH, similarly, is the reciprocal of absolute risk increase for an adverse outcome, such as accidental purchases or complaints. Reporting both side by side shows whether benefits outweigh harms, and confidence intervals around them highlight uncertainty. This brings ethical clarity to experiments with dark-pattern risks and helps stakeholders reason about impact in human terms.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. Identify the benefit outcome and any harm outcomes for an intervention.
  2. Compute absolute risk differences between treatment and control for each, with confidence intervals.
  3. Convert to NNT and NNH (1 / absolute difference) and their intervals, handling intervals that cross zero carefully.
  4. Express results as 'for every X users exposed, one gains benefit; for every Y, one is harmed'.
  5. Weigh benefits against harms, considering severity.
  6. Include NNT/NNH in experiment reports and decision memos.

Sources

  • Laupacis, Sackett & Roberts, An assessment of clinically useful measures of the consequences of treatment (1988), New England Journal of Medicine
  • Altman, Confidence intervals for the number needed to treat (1998), BMJ

Also known as

  • NNT
  • NNH
  • Number needed to harm
  • Absolute risk reduction

Categories

  • Epidemiology & Population Health