Description
Measure how many bits of uncertainty about one variable are removed, on average, by knowing another, as I(X;Y) = H(X) - H(X|Y), to quantify dependence between categorical variables.
Mutual information extends entropy from one variable to a pair. MacKay defines the joint entropy H(X,Y), the conditional entropy H(X|y) for one value of y and H(X|Y) averaged over y, the chain rule H(X,Y) = H(X) + H(Y|X), and the mutual information I(X;Y) = H(X) - H(X|Y), which is symmetric and never negative. It is the average reduction in uncertainty about x from learning y. Conditional mutual information I(X;Y|Z) is computed within each value of z and averaged, which helps when a third variable may confound. Unlike correlation it captures any form of dependence among categorical values and is measured in bits. MacKay warns that learning one particular value of y can increase uncertainty about x even though the average decreases, and that Venn-diagram pictures of three-variable entropies mislead.
Attributes
- Discipline
- Information Theory, Physics & Mathematics
- Loss type
- Unmigrated
- Stage
- Discovery, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Tabulate the joint distribution of two categorical variables from counts, for example entry page by task outcome.
- Compute the marginal entropies H(X) and H(Y) in bits.
- Compute the conditional entropy H(X|Y) by averaging H(X|y) over y weighted by P(y).
- Compute I(X;Y) = H(X) - H(X|Y), equivalently H(X) + H(Y) - H(X,Y).
- Where a third variable may confound, compute I(X;Y|Z) within levels of Z and average.
- With small tables, correct for bias or compare with values from shuffled data before interpreting.
Sources
- Information Theory, Inference, and Learning Algorithms (MacKay, 2003)
Also known as
- Information gain
- Conditional entropy
- Uncertainty reduction
- I(X;Y)
Categories
- Information Theory, Physics & Mathematics