Description
A calibration test for binary regression that groups cases into deciles of predicted risk and compares observed with expected event counts in each group.
With continuous covariates almost every case has its own covariate pattern, so the Pearson and deviance statistics cannot be referred to chi-square. Hosmer and Lemeshow restore large expected counts by sorting cases on fitted probability and cutting them into about ten deciles of risk. Observed and expected events and non-events in each group form a g x 2 table whose Pearson statistic C is referred to chi-square on g-2 degrees of freedom (g in an external validation sample). The table itself shows where the model over- or under-predicts. The authors recommend pairing C with the Osius-Rojek normal approximation and Stukel's two-df test of the logistic tails, and stress that calibration and discrimination (ROC area) are separate properties to be reported together. Power is low below about 400 cases, and software tie-handling can change C.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Unmigrated
- Stage
- Post-launch, Evaluation
- Data
- Quantitative
- Effort
- Hours
Steps
- Fit the binary model and save each case's fitted probability.
- Sort cases by fitted probability and split them into g = 10 groups at the percentiles (deciles of risk) rather than at fixed cut-points.
- In each group sum observed events and fitted probabilities (expected events), and do the same for non-events.
- Compute the Pearson chi-square C over the g x 2 table and refer it to chi-square on g-2 df, or on g df when testing in an external sample.
- Inspect observed versus expected counts per decile, merging adjacent groups where expected counts are very small; treat fewer than 6 groups as uninformative.
- Supplement C with the Osius-Rojek statistic and Stukel's score test, then examine case-level diagnostics.
Sources
- Applied Logistic Regression (Hosmer & Lemeshow, 2000)
Also known as
- Deciles-of-risk test
- Hosmer-Lemeshow C statistic
- Hosmer-Lemeshow test
- Osius-Rojek test
- Stukel test
Categories
- Analytics & Behaviour
- Statistics & Modelling