Within-Group Agreement Index (rwg)

Description

Quantifies how far members of the same group agree in their ratings, compared with the spread expected if they answered at random, before or alongside averaging their answers to the group level.

When individuals rate a shared property, such as their team's capability or their company's experience of a product, researchers often average the ratings to get one group score. The rwg index estimates within-group agreement by comparing the observed variance of ratings with the variance expected if responses were random. With multiple parallel items, rwg(J) combines the mean observed item variance with the number of items J; values near 1 mean strong agreement and values near 0 mean agreement no better than chance. The expected variance is usually taken from a uniform distribution, (A squared minus 1) divided by 12, where A is the number of response categories. It can be adjusted when response biases such as leniency or social desirability are expected. The Handbook warns against several common practices. The often-quoted 0.70 cut-off has no clear source. Too many response categories inflate rwg. Groups should not be dropped for low agreement. Instead, rwg should be reported descriptively and used as a group-level variable in multilevel models. For UX, this matters in B2B research, where scores from several users at one customer account are averaged into an account-level satisfaction or health score.

Attributes

Loss type
Siloed
Effort
Hours

Steps

  1. Confirm that the items measure a shared, group-level property and form one dimension.
  2. Collect ratings from several members of each group, such as each customer account or team.
  3. Choose the expected random variance (uniform by default, adjusted if a response bias is expected) and state the choice.
  4. Compute rwg(J) for each group across the parallel items.
  5. Report the distribution of rwg values rather than discarding low-agreement groups.
  6. Use rwg as a moderator in multilevel models, or flag accounts with low agreement for qualitative follow-up.

Sources

  • Tenenbaum, G. & Eklund, R. (eds), Handbook of Sport Psychology, 3rd ed. (2007)

Also known as

  • rwg
  • rwg(J)
  • Interrater agreement for aggregation
  • James, Demaree and Wolf agreement index
  • Within-team consensus index

Categories

  • Analytics & Behaviour
  • Statistics & Modelling