Description
A heuristic evaluation in which the severity of each issue is adjusted according to how strongly that kind of issue affects critics' review scores, so fixes are prioritised by likely impact on reception.
Critic-proofing builds on the classic heuristic analysis process but changes how problems are prioritised. Before evaluating the game, the researcher conducts a baseline analysis of a set of video game reviews to learn which types of problem, mapped to the heuristics, critics mention and how strongly they are associated with lower scores. During the evaluation, each issue found against a heuristic is then given a severity that reflects not only its effect on the player but the effect that kind of issue tends to have on a critic's review. The aim is to make sure that problems most likely to hurt reception are fixed first. Livingston notes that the process can be extended to any set of heuristics, provided the researcher is willing to do the baseline review analysis. For UX teams beyond games, the same principle applies wherever public ratings matter, for example app store reviews or analyst evaluations: severity can be weighted by evidence of what reviewers penalise rather than by expert judgement alone.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Latent
- Goal
- Prioritise, Evaluate usability
- Stage
- Evaluation
- Data
- Mixed
- Effort
- Days
Steps
- Choose the heuristic set to be used for evaluation.
- Collect a baseline sample of reviews for comparable products and code which heuristic categories they criticise.
- Estimate how strongly each category of problem is associated with lower review scores.
- Carry out the heuristic evaluation of your product in the usual way.
- Adjust each issue's severity using the review-based weighting for its heuristic category.
- Prioritise and report fixes by the adjusted severity.
Sources
- Drachen, A., Mirza-Babaei, P. & Nacke, L. (eds), Games User Research (2018)
Also known as
- Critic-proof heuristic analysis
- Review-weighted severity rating
Categories
- User Research
- Usability & Evaluation