Whole-Call Recording and Reconstructed Conversation Review

Description

Listen to complete, randomly sampled conversations, recorded or stitched together from logged prompts and utterances, to find dialogue-level problems invisible in per-turn metrics.

Most voice and conversational interface analysis works per utterance: recognition accuracy, no-match rates, intent confidence. Some problems only make sense in context, such as a failure ten turns in caused by an earlier misunderstanding. IVR practice therefore recorded whole calls, system and user, for a subset of traffic. A cheaper alternative reconstructs each conversation by stitching logged system prompts and user utterances in order, adding barge-in timing where enabled and screen events for multimodal apps. Calls are sampled randomly so the set is neither only smooth nor only failing, across at least a week because user types vary by time of day and weekday. Tools may need building: a turn-by-turn view of what the system said, what the user said and what happened next.

Attributes

Loss type
Siloed
Stage
Post-launch
Effort
Days

Steps

  1. Record whole calls for a subset of traffic, or log prompts, utterances, barge-in timing and screen events so conversations can be reconstructed.
  2. Randomly sample conversations across at least one full week, avoiding samples of only errors or only successes.
  3. Anonymise audio and transcripts before review.
  4. Listen to (and for multimodal apps, watch) each conversation turn by turn, noting where and why the dialogue goes wrong in context.
  5. Code dialogue-level issues such as misleading prompts, error-recovery loops and abandoned goals.
  6. Feed issues into prompt redesign, dialogue flow changes and recognition tuning, then resample to check.

Sources

  • Designing Voice User Interfaces (Pearl, 2017)

Also known as

  • Whole call recording
  • Conversation reconstruction review
  • Call listening

Categories

  • User Research
  • Qualitative Analysis & Synthesis