Verbal Protocol Analysis

Description

Rigorous coding of think-aloud transcripts with a theory-derived scheme to identify cognitive operations, strategies and mental models, with reliability checks.

Verbal protocol analysis treats think-aloud transcripts as data about cognitive processes. Transcripts are segmented into units, typically one idea or utterance, and each unit is coded with a scheme derived from a cognitive model of the task, for example goal setting, information retrieval, hypothesis generation, evaluation or error recovery. Coded sequences reveal strategies, the order of operations, where knowledge is missing and how mental models guide action. This is far more than extracting quotes for themes. Rigour comes from a coding scheme defined before analysis and grounded in theory, independent double coding with inter-rater agreement (for example Cohen's kappa of 0.7 or higher), and analysis of frequencies and transitions between codes, often comparing groups such as novices and experts.

Attributes

Loss type
Vestigial
Effort
Weeks

Steps

  1. Collect concurrent think-aloud data on representative tasks following Ericsson and Simon's instructions.
  2. Transcribe verbatim and segment transcripts into utterance-level units.
  3. Develop a coding scheme from a cognitive model of the task and pilot it on one transcript.
  4. Have two coders independently code at least 20 percent of the data and compute inter-rater agreement, refining the scheme until kappa is at least 0.7.
  5. Code the full dataset.
  6. Analyse code frequencies, sequences and transitions, comparing participant groups.
  7. Interpret findings in terms of strategies, knowledge gaps and mental models.

Sources

  • Lost Methods in UX
  • Cognitive Psychology
  • Protocol Analysis (Ericsson & Simon)
  • Lost Methods in UX — A Complete Treatise (Gopinath)
  • Ericsson KA & Simon HA (1984; rev. 1993) Protocol Analysis: Verbal Reports as Data. MIT Press
  • Boren T & Ramey J (2000) Thinking aloud: reconciling theory and practice. IEEE Trans Prof Comm
  • The Sage Handbook of Qualitative Data Analysis — Flick (ed.)
  • Sage Handbook of Qualitative Data Analysis (Flick)
  • Rapid Contextual Design (Holtzblatt)
  • Practice of Qualitative Research
  • Qualitative Data Analysis for Applied Policy Research (Ritchie & Spencer)
  • Qualitative Research Practice (Ritchie & Lewis)
  • Rapid Contextual Design (Morgan Kaufmann)
  • Error Taxonomy Playbook (Gopinath)
  • The Inquiry Series, Ch 34 — Framework Analysis & Systematic Qualitative Synthesis
  • Protocol Analysis: Verbal Reports as Data (Ericsson & Simon)
  • Handbook of Cognitive Task Design (Hollnagel)
  • Human Problem Solving (Newell & Simon)
  • Tenenbaum, G. & Eklund, R. (eds), Handbook of Sport Psychology, 3rd ed. (2007)

Also known as

  • VPA
  • Coded protocol analysis
  • Protocol analysis
  • Ericsson-Simon think-aloud
  • Concurrent verbalisation analysis
  • Framework method
  • Matrix-based qualitative analysis
  • Ritchie and Spencer framework
  • Charting
  • Cases by themes matrix
  • NatCen framework approach
  • Framework analysis
  • Verbal protocol analysis
  • Ericsson and Simon think-aloud
  • Concurrent verbalisation
  • Ericsson-Simon protocol analysis
  • Cognitive protocol analysis
  • Cases-by-themes matrix
  • Framework Analysis (Ritchie and Spencer)

Categories

  • User Research
  • Cognitive Psychology & Perception