Description
Reads and classifies chatbot or voice-interface logs turn by turn, locating where conversations fail, then tunes prompts, models and thresholds and regression-tests the changes.
Conversational interfaces leave a complete transcript of every session, a direct record of what users try to do. The rigorous form comes from IVR tuning: log every turn by design, including recognition result and confidence, N-best list, audio, state name, no-speech timeouts, no-matches, latency and barge-in. Humans transcribe a sample of utterances because the recogniser cannot measure its own accuracy. In-grammar utterances are classified as correct accept, false accept or false reject, out-of-grammar ones as correct reject or false accept, separating out-of-domain from in-domain failures. Per-state indicators (dropout, silence, repeats, back, modality) locate problems; thresholds, timeouts, vocabularies and prompt wording are tuned; and recorded audio is rerun to confirm improvement. Chatbot practice adds routing unmatched inputs to editors and logging success and failure events with their place in the dialogue.
Attributes
- Discipline
- Product Analytics & Strategy
- Loss type
- Vestigial
- Stage
- Evaluation, Post-launch
- Data
- Mixed
- Effort
- Weeks
Steps
- Specify logging of every turn with developers from the start: input, recognised intent or result, confidence, state or node, errors, latency and outcome; anonymise transcripts and audio.
- Start by reading raw transcripts end to end to learn how users actually phrase requests.
- Hand-transcribe or label a sample of utterances and classify each as correct accept, false accept, false reject or correct reject, separating in-domain from out-of-domain inputs.
- Review per-state indicators such as dropout, no-match and timeout rates, repeats, back and help requests, and escalation to a human.
- Route unmatched inputs to a shared queue where editors review them weekly and extend intents or flows, prioritising frequent signals over fringe requests.
- Tune confidence thresholds, timeouts, vocabularies, intents and prompt wording; in pilots, set a goal such as 75% of utterances handled.
- Regression-test by rerunning recorded inputs through the tuned system to confirm error rates fall without new breakage; repeat continuously.
Sources
- Designing Bots (Shevat, 2017)
- Designing Voice User Interfaces (Pearl, 2017)
- Chatbot Research and Design: CONVERSATIONS 2022 (Følstad et al., 2023)
Also known as
- Bot transcript and log review
- Conversation log review
- Chatbot transcript reading
- Unmatched-input review
- VUI performance tuning
- Speech application tuning
- Recognition tuning
- In-grammar/out-of-grammar analysis
Categories
- Analytics & Behaviour
- Product Analytics & Strategy