Conversation Log Review and Tuning

Description

Reads and classifies chatbot or voice-interface logs turn by turn, locating where conversations fail, then tunes prompts, models and thresholds and regression-tests the changes.

Conversational interfaces leave a complete transcript of every session, a direct record of what users try to do. The rigorous form comes from IVR tuning: log every turn by design, including recognition result and confidence, N-best list, audio, state name, no-speech timeouts, no-matches, latency and barge-in. Humans transcribe a sample of utterances because the recogniser cannot measure its own accuracy. In-grammar utterances are classified as correct accept, false accept or false reject, out-of-grammar ones as correct reject or false accept, separating out-of-domain from in-domain failures. Per-state indicators (dropout, silence, repeats, back, modality) locate problems; thresholds, timeouts, vocabularies and prompt wording are tuned; and recorded audio is rerun to confirm improvement. Chatbot practice adds routing unmatched inputs to editors and logging success and failure events with their place in the dialogue.

Attributes

Loss type
Vestigial
Data
Mixed
Effort
Weeks

Steps

  1. Specify logging of every turn with developers from the start: input, recognised intent or result, confidence, state or node, errors, latency and outcome; anonymise transcripts and audio.
  2. Start by reading raw transcripts end to end to learn how users actually phrase requests.
  3. Hand-transcribe or label a sample of utterances and classify each as correct accept, false accept, false reject or correct reject, separating in-domain from out-of-domain inputs.
  4. Review per-state indicators such as dropout, no-match and timeout rates, repeats, back and help requests, and escalation to a human.
  5. Route unmatched inputs to a shared queue where editors review them weekly and extend intents or flows, prioritising frequent signals over fringe requests.
  6. Tune confidence thresholds, timeouts, vocabularies, intents and prompt wording; in pilots, set a goal such as 75% of utterances handled.
  7. Regression-test by rerunning recorded inputs through the tuned system to confirm error rates fall without new breakage; repeat continuously.

Sources

  • Designing Bots (Shevat, 2017)
  • Designing Voice User Interfaces (Pearl, 2017)
  • Chatbot Research and Design: CONVERSATIONS 2022 (Følstad et al., 2023)

Also known as

  • Bot transcript and log review
  • Conversation log review
  • Chatbot transcript reading
  • Unmatched-input review
  • VUI performance tuning
  • Speech application tuning
  • Recognition tuning
  • In-grammar/out-of-grammar analysis

Categories

  • Analytics & Behaviour
  • Product Analytics & Strategy