Description
Tests whether usage, content popularity or query frequency follows a power law, with proper estimation and comparison against alternative heavy-tailed distributions.
Many product phenomena are heavy-tailed: a few users generate most activity, a few queries dominate search, a few items get most views. Claims of 'power laws' are often made from straight-ish lines on log-log plots, which is unreliable. Clauset, Shalizi and Newman's method estimates the exponent by maximum likelihood, selects the lower cutoff x_min via Kolmogorov-Smirnov distance, tests goodness of fit with bootstrapping, and compares the power law against log-normal, exponential and stretched exponential alternatives with likelihood ratio tests. Getting the distribution right matters: it determines whether averages are meaningful, how to sample users for research and how to design for the head versus the long tail.
Attributes
- Discipline
- Information Theory, Physics & Mathematics
- Loss type
- Vestigial
- Goal
- Measure behaviour
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Collect the quantity (e.g. sessions per user, views per item, queries per term).
- Plot the complementary cumulative distribution on log-log axes for exploration only.
- Estimate x_min and the exponent alpha by maximum likelihood (e.g. powerlaw Python package).
- Run bootstrap goodness-of-fit tests (p above 0.1 supports plausibility).
- Compare against log-normal and other alternatives with likelihood ratio tests.
- Interpret implications: concentration, sampling strategies, and design for head and tail.
Sources
- Clauset, Shalizi & Newman, Power-law distributions in empirical data (2009), SIAM Review
- Newman, Power laws, Pareto distributions and Zipf's law (2005), Contemporary Physics
Also known as
- Power-law fitting
- Zipf analysis
- Heavy-tail analysis
- Clauset-Shalizi-Newman method
Categories
- Information Theory, Physics & Mathematics