Distribution-Free Confidence Intervals for Percentiles

Description

Bracket any population percentile, such as the median or 90th percentile task time, between two sorted sample values chosen by a binomial rule, with no distributional assumption.

UX metrics such as task time, load time and spend are skewed, and the percentile of interest (median, 90th, 95th) is often reported without any interval. For a continuous measure, the interval from the r-th to the s-th smallest observation covers the population p-th percentile with a probability given by a binomial sum in r, s, n and p, whatever the shape of the distribution. Choose r and s to reach the target confidence with the narrowest span; for the median use symmetric ranks, which links the interval to the sign test. A quick rule for the median counts about half of root n times the normal critical value either side of the sample median. Because coverage moves in steps, interpolation between neighbouring order statistics brings it close to nominal. Extensions cover differences of percentiles, simultaneous intervals and two-sample median differences.

Attributes

Loss type
Latent
Effort
Hours

Steps

  1. Choose the percentile p (for example 0.5, 0.9 or 0.95) and confidence level (for example 95 percent).
  2. Sort the n observations.
  3. For candidate rank pairs (r, s), compute the binomial probability that between r and s - 1 observations fall below the p-th percentile.
  4. Pick the pair with coverage at least the target and the smallest gap; use symmetric ranks for the median (with n = 100 at 95 percent, roughly the 40th to 61st values).
  5. Optionally interpolate between adjacent order statistics (Hettmansperger-Sheather) to approach the nominal level.
  6. Report the interval as the r-th to s-th ordered values and state the achieved coverage; note when n is too small to support the requested extreme percentile.

Sources

  • Order Statistics (David & Nagaraja, 2003)

Also known as

  • Order-statistic confidence interval for a quantile
  • Nonparametric percentile confidence interval
  • Binomial confidence interval for the median
  • Thompson interval
  • Hettmansperger-Sheather interpolated interval
  • Distribution-free confidence intervals for percentiles from order statistics

Categories

  • Analytics & Behaviour
  • Statistics & Modelling