Kernel Density Estimation

Description

Estimate the shape of a distribution by averaging a small kernel centred on each observation, choosing the bandwidth so the estimate neither hides structure nor shows noise.

Kernel density estimation places a window function, usually Gaussian, on every observation and averages them, so each point contributes to the estimated density according to its distance. The window must itself be a density so the estimate integrates to one. The bandwidth h controls everything: too wide and the estimate is smooth but loses structure such as bimodality; too narrow and it becomes a series of spikes at the data points. Duda, Hart and Stork show widths shrinking with the square root of n and point to cross-validation for choosing h; Bishop derives the histogram, kernel and k-nearest-neighbour estimators from the same idea. The k-nearest-neighbour estimate lets the region grow until it holds k points, adapting resolution to local density. Per-group densities can also be compared or turned into a classifier.

Attributes

Loss type
Siloed
Effort
Hours

Steps

  1. Collect the observations for one or more variables.
  2. Choose a kernel that is itself a density, typically Gaussian.
  3. Choose a bandwidth by a rule of thumb or by cross-validation, and try at least one wider and one narrower value.
  4. Evaluate the estimate on a grid by averaging the kernels centred on every observation.
  5. Inspect for over-smoothing (lost modes) or under-smoothing (spiky noise) and settle the bandwidth.
  6. Optionally estimate densities per group and compare them, or use a k-nearest-neighbour estimate where data density varies widely.

Sources

  • Pattern Classification (Duda et al., 2001)
  • Pattern Recognition and Machine Learning (Bishop, 2006)
  • Introduction to Nonparametric Regression (Takezawa, 2006)
  • Loss Models: From Data to Decisions (Klugman et al., 2012)

Also known as

  • Kernel density estimation (Parzen windows) with window-width choice
  • Parzen-window density estimation
  • KDE
  • Nonparametric density estimation
  • K-nearest-neighbour density estimation

Categories

  • Analytics & Behaviour
  • Statistics & Modelling