Description
Tests whether respondents from different groups with the same level of the measured trait answer an item differently, so biased items can be found, modelled or removed before comparing groups.
An item shows DIF when two people from different groups at the same trait level have different probabilities of each response. DIF items distort group differences and correlations, so a gap between countries, devices or segments may come from wording or translation rather than real differences. Classical methods (Mantel-Haenszel, logistic or ordinal logistic regression conditioning on total score) are easy but sample specific. IRT methods give results that generalise, graphical diagnostics (group trace lines, expected item and test scores) and DIF-adjusted scores for sensitivity analysis. The illustrated IRT route calibrates both groups simultaneously and compares nested models with likelihood-ratio tests, first for any DIF and then for slope and threshold differences. Groups are linked through anchor items assumed DIF-free; one can suffice but three or more is better, chosen from prior knowledge, random assignment or iterative purification.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Siloed
- Stage
- Evaluation, Discovery
- Data
- Quantitative
- Effort
- Weeks
Steps
- Confirm the scale is suitable for unidimensional IRT (or choose a classical method if samples are small).
- Start with all other items as a temporary anchor and test each item in turn at a liberal alpha (.05) so that the final anchor set is conservative.
- For each studied item, compare a model with its parameters free across groups against one with them constrained equal; a chi-square difference above 3.84 (1 df) flags DIF.
- For flagged items, test slope equality and then threshold equality to locate the type of DIF.
- Purify the anchor set by removing DIF items and retest the remaining items.
- Plot group trace lines and expected scores, compute DIF-adjusted scale scores and compare them with raw scores to judge practical impact before deciding to drop, rewrite or model the items.
Sources
- The SAGE Handbook of Quantitative Methods in Psychology (Millsap & Maydeu-Olivares, 2009)
Also known as
- Item bias analysis
- IRT likelihood-ratio DIF test
- Mantel-Haenszel DIF
- Logistic regression DIF
- DFIT
Categories
- Analytics & Behaviour
- Survey & Psychometrics