Mutual Information Between Two Variables

Description

Measure how many bits of uncertainty about one variable are removed, on average, by knowing another, as I(X;Y) = H(X) - H(X|Y), to quantify dependence between categorical variables.

Mutual information extends entropy from one variable to a pair. MacKay defines the joint entropy H(X,Y), the conditional entropy H(X|y) for one value of y and H(X|Y) averaged over y, the chain rule H(X,Y) = H(X) + H(Y|X), and the mutual information I(X;Y) = H(X) - H(X|Y), which is symmetric and never negative. It is the average reduction in uncertainty about x from learning y. Conditional mutual information I(X;Y|Z) is computed within each value of z and averaged, which helps when a third variable may confound. Unlike correlation it captures any form of dependence among categorical values and is measured in bits. MacKay warns that learning one particular value of y can increase uncertainty about x even though the average decreases, and that Venn-diagram pictures of three-variable entropies mislead.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. Tabulate the joint distribution of two categorical variables from counts, for example entry page by task outcome.
  2. Compute the marginal entropies H(X) and H(Y) in bits.
  3. Compute the conditional entropy H(X|Y) by averaging H(X|y) over y weighted by P(y).
  4. Compute I(X;Y) = H(X) - H(X|Y), equivalently H(X) + H(Y) - H(X,Y).
  5. Where a third variable may confound, compute I(X;Y|Z) within levels of Z and average.
  6. With small tables, correct for bias or compare with values from shuffled data before interpreting.

Sources

  • Information Theory, Inference, and Learning Algorithms (MacKay, 2003)

Also known as

  • Information gain
  • Conditional entropy
  • Uncertainty reduction
  • I(X;Y)

Categories

  • Information Theory, Physics & Mathematics