Description
Estimate the shape of a distribution by averaging a small kernel centred on each observation, choosing the bandwidth so the estimate neither hides structure nor shows noise.
Kernel density estimation places a window function, usually Gaussian, on every observation and averages them, so each point contributes to the estimated density according to its distance. The window must itself be a density so the estimate integrates to one. The bandwidth h controls everything: too wide and the estimate is smooth but loses structure such as bimodality; too narrow and it becomes a series of spikes at the data points. Duda, Hart and Stork show widths shrinking with the square root of n and point to cross-validation for choosing h; Bishop derives the histogram, kernel and k-nearest-neighbour estimators from the same idea. The k-nearest-neighbour estimate lets the region grow until it holds k points, adapting resolution to local density. Per-group densities can also be compared or turned into a classifier.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Goal
- Synthesise data, Segment users
- Stage
- Discovery, Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Collect the observations for one or more variables.
- Choose a kernel that is itself a density, typically Gaussian.
- Choose a bandwidth by a rule of thumb or by cross-validation, and try at least one wider and one narrower value.
- Evaluate the estimate on a grid by averaging the kernels centred on every observation.
- Inspect for over-smoothing (lost modes) or under-smoothing (spiky noise) and settle the bandwidth.
- Optionally estimate densities per group and compare them, or use a k-nearest-neighbour estimate where data density varies widely.
Sources
- Pattern Classification (Duda et al., 2001)
- Pattern Recognition and Machine Learning (Bishop, 2006)
- Introduction to Nonparametric Regression (Takezawa, 2006)
- Loss Models: From Data to Decisions (Klugman et al., 2012)
Also known as
- Kernel density estimation (Parzen windows) with window-width choice
- Parzen-window density estimation
- KDE
- Nonparametric density estimation
- K-nearest-neighbour density estimation
Categories
- Analytics & Behaviour
- Statistics & Modelling