Description
Extends the Rasch model beyond persons and items to further facets such as raters, tasks or occasions, so each can be placed on one scale and rater severity can be measured and corrected.
Standard Rasch models treat data as a two-way matrix of persons by items. The many-facet Rasch model adds further facets, such as the raters who judge responses, the tasks, the occasion or person characteristics, giving a structure of persons by items by raters and so on. Each element is estimated on a common logit scale: person ability, item difficulty, rater severity and task difficulty. Fit statistics show which raters are inconsistent, and interaction terms show bias, for example a rater who is harsh only on one group. The Handbook describes uses in sport and rehabilitation. These include recalibrating a gross motor test with eight facets, analysing judged sports, where the model has exposed unfair ice-skating judges, and a two-facet analysis of the Borg scale showing it reflects cumulative exertion. For UX the model fits any design where several judges rate several things: expert reviews and severity ratings of usability problems, human evaluation of AI or search output, design-critique scoring, accessibility audits and graded usability tasks. It gives fair scores that are corrected for which rater happened to judge which item, and shows which raters need recalibration.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Define the facets in your rating design, for example participants or outputs, criteria, raters and tasks.
- Make sure the design links facets (raters overlap on some items), because disconnected subsets cannot be placed on one scale.
- Collect ratings on a consistent ordinal scale.
- Fit the model with Rasch software that supports facets, and check category functioning.
- Inspect rater severity and fit, and bias interactions between raters and other facets.
- Report fair, severity-adjusted scores and give feedback to misfitting raters.
Sources
- Tenenbaum, G. & Eklund, R. (eds), Handbook of Sport Psychology, 3rd ed. (2007)
Also known as
- MFRM
- Many-faceted Rasch model
- FACETS analysis
- Linacre facets model
- Rater-adjusted Rasch scaling
Categories
- Analytics & Behaviour
- Statistics & Modelling