Experience annotation formats (rating, class, preference)

Description

Labels experience by players or observers as a rating, a class choice or a preference between variants, preferring ranked comparisons because ratings suffer order and inconsistency effects.

Player-experience models need ground-truth labels of experience. Annotations come either from the player (free response during play, or forced questionnaire items) or from third parties (experts, designers, crowd workers) who triangulate behaviour, head pose and context using schemes such as FACS. Three formats exist. Rating gives a scalar or vector on a scale, as in the Game Experience Questionnaire. Class asks for a choice of state, from a yes/no question to a position on the Geneva Emotion Wheel. Preference asks which of two or more variants or sessions was more engaging, frustrating or fun. Yannakakis and Hallam found ratings prone to order-of-play and inconsistency effects that ranked comparisons avoid, because people judge relative differences more reliably than absolute levels. The choice of format therefore shapes the reliability of any model trained on the labels.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Days

Steps

  1. Decide who annotates: the user (self-report) or trained observers or crowd workers (third person), with shared guidelines for the latter.
  2. Choose free response or forced items.
  3. Choose the format: rating scale, class label, or preference between variants or sessions.
  4. For preference, present two or more variants in counterbalanced order and ask which was more engaging, frustrating or enjoyable.
  5. Check consistency (for example intransitive choices) and inter-annotator agreement for third-person labels.
  6. Use the annotations as outputs for experience models or as comparison data, modelling ranks with preference-learning methods.

Sources

  • Player Modeling (Yannakakis et al., 2013)

Also known as

  • Experience annotation formats: rating, class and preference (rank) based
  • Pairwise preference self-report
  • Rank-based experience annotation
  • Rating vs preference self-reporting
  • Third-person experience annotation

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics