Description
Listen to complete, randomly sampled conversations, recorded or stitched together from logged prompts and utterances, to find dialogue-level problems invisible in per-turn metrics.
Most voice and conversational interface analysis works per utterance: recognition accuracy, no-match rates, intent confidence. Some problems only make sense in context, such as a failure ten turns in caused by an earlier misunderstanding. IVR practice therefore recorded whole calls, system and user, for a subset of traffic. A cheaper alternative reconstructs each conversation by stitching logged system prompts and user utterances in order, adding barge-in timing where enabled and screen events for multimodal apps. Calls are sampled randomly so the set is neither only smooth nor only failing, across at least a week because user types vary by time of day and weekday. Tools may need building: a turn-by-turn view of what the system said, what the user said and what happened next.
Attributes
- Discipline
- Qualitative Analysis & Synthesis
- Loss type
- Siloed
- Stage
- Post-launch
- Data
- Qualitative
- Effort
- Days
Steps
- Record whole calls for a subset of traffic, or log prompts, utterances, barge-in timing and screen events so conversations can be reconstructed.
- Randomly sample conversations across at least one full week, avoiding samples of only errors or only successes.
- Anonymise audio and transcripts before review.
- Listen to (and for multimodal apps, watch) each conversation turn by turn, noting where and why the dialogue goes wrong in context.
- Code dialogue-level issues such as misleading prompts, error-recovery loops and abandoned goals.
- Feed issues into prompt redesign, dialogue flow changes and recognition tuning, then resample to check.
Sources
- Designing Voice User Interfaces (Pearl, 2017)
Also known as
- Whole call recording
- Conversation reconstruction review
- Call listening
Categories
- User Research
- Qualitative Analysis & Synthesis