Chi-Square and Fisher's Exact Tests

Description

Tests whether categorical outcomes differ across groups, such as preference by device or success by design.

The chi-square test compares observed counts in a contingency table with the counts expected if the variables were unrelated. Fisher's exact test computes the exact probability for small tables. Bruce and Bruce present a resampling approach that shuffles labels to build the null distribution directly, which makes the logic intuitive. For 2×2 tables Sauro and Lewis recommend the N-1 chi-square over both the classic test and Fisher's test; for larger tables the standard chi-square applies when expected counts are adequate.

Attributes

Loss type
Active
Stage
Evaluation
Effort
Hours

Steps

  1. Build the contingency table of counts.
  2. Check expected counts; for small 2×2 tables use the N-1 variant, for tiny tables use Fisher's exact test.
  3. Compute the statistic and p-value (or simulate by permutation).
  4. Inspect standardised residuals to see which cells drive the result.
  5. Report counts, percentages and an effect size such as Cramér's V.

Sources

  • Practical Statistics for Data Scientists (Bruce & Bruce)
  • Quantifying the User Experience (Sauro & Lewis)
  • Companion Book to Quantifying the User Experience (Lewis & Sauro)
  • Myatt, G., Making Sense of Data: A Practical Guide to Exploratory Data Analysis and Data Mining (2007)
  • Gratton, C. & Jones, I., Research Methods for Sports Studies (2010)
  • Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)

Also known as

  • Pearson chi-square test
  • Fisher-Irwin test
  • Test of independence

Categories

  • Analytics & Behaviour
  • Statistics & Modelling