Review Analysis Methodology (RVA)

Description

A grounded theory analysis of a stratified sample of professional reviews, coded by several researchers against focus questions, that produces a theoretical framework of how and why a product is described as it is.

After launch, everyone on a team reads reviews and forms opinions, usually with bias and without method. RVA turns reviews into rigorous evidence. Livingston treats professional reviewers as highly articulate expert players who also place a game in its wider cultural context. The classic study uses two to six researchers, two to four weeks, six to eight focus questions and 20 to 30 reviews sampled from the top, middle and bottom thirds of an aggregator's score range, so that a breadth of opinion is captured. Reviews are saved as plain text and open-coded with the focus questions as loose direction; after about half are coded, the team switches to selective coding with three codes per idea: focus question, conceptual label and positive or negative sentiment. Each review is memoed. Peer debriefs at roughly 25, 50, 75 and 100 per cent of coding merge codes, sort them into concepts and categories and challenge emerging theory, while negative case analysis seeks reviews that contradict it. Only after the theory is formed is a frequency count of codes and sentiment made, to communicate weight. The report answers each focus question with quotes and counts. The method transfers to app store reviews, analyst reports or expert press for any product.

Attributes

Loss type
Siloed
Stage
Post-launch
Effort
Weeks

Steps

  1. Plan the study: two or more researchers, six to eight focus questions and a time budget of two to four weeks.
  2. Sample 20 to 30 reviews across the top, middle and bottom thirds of scores and save them as plain text.
  3. Open-code the first half of the reviews, guided loosely by the focus questions, and write a memo for each.
  4. Hold peer debriefs to merge codes, sort concepts into categories and develop the theory; then selectively code the rest with focus question, concept and sentiment codes.
  5. Run negative case analysis to find reviews that contradict the emerging framework, until saturation.
  6. After the theory is articulated, count codes and sentiment per category.
  7. Report how the theory answers each focus question, supported by quotes and frequencies, with actionable implications.

Sources

  • Drachen, A., Mirza-Babaei, P. & Nacke, L. (eds), Games User Research (2018)

Also known as

  • RVA
  • Game review analysis
  • Grounded theory review analysis

Categories

  • User Research
  • Qualitative Analysis & Synthesis