Description
Decomposes measurement variance into multiple sources (participants, tasks, raters, occasions) and designs studies that reach target reliability efficiently.
Classical reliability treats error as one undifferentiated term. G-theory uses ANOVA-style variance components to estimate how much score variation comes from the object of measurement (e.g. designs) versus facets such as tasks, raters, participants and occasions, and their interactions. A G-study estimates these components; a D-study then asks how reliability (generalizability coefficient) would change with different numbers of tasks, raters or participants. For UX benchmarks and heuristic evaluations, this tells you whether to add evaluators, tasks or participants to get dependable comparisons, and reveals when results mostly reflect which tasks were chosen.
Attributes
- Discipline
- Survey & Psychometrics
- Loss type
- Unmigrated
- Stage
- Evaluation
- Data
- Quantitative
- Effort
- Days
Steps
- Define the object of measurement (e.g. product versions) and facets (tasks, participants, raters, occasions).
- Collect data in a crossed design where possible (every rater rates every design on every task).
- Estimate variance components with ANOVA or mixed models (e.g. gtheory or lme4).
- Compute generalizability (relative) and dependability (absolute) coefficients.
- Run D-studies varying the number of conditions per facet to find efficient designs reaching e.g. 0.8.
- Report variance components so others can see which facets drive error.
Sources
- Cronbach et al., The Dependability of Behavioral Measurements (1972)
- Brennan, Generalizability Theory (2001)
- Shavelson & Webb, Generalizability Theory: A Primer (1991)
Also known as
- G-theory
- G-study and D-study
- Variance components reliability
Categories
- Analytics & Behaviour
- Survey & Psychometrics