Scripted Speaker Recognition Testing

Description

Before launch, 10 to 20 people read a fixed script of utterances to a voice system to tune timeouts and catch common misrecognitions.

A pre-launch check from interactive voice response practice that base recognition parameters are right before real users are exposed. A small set of speakers, typically 10 to 20, read a predetermined script covering the key utterance types: digit strings, names, domain terms and expected commands. Results are compared with the script to tune end-of-speech timeouts (phone numbers need longer pauses between digit groups) and to identify frequent misrecognitions, which are added to expected phrases or grammars. Even with cloud recognisers it catches domain-specific failures; Pearl's medical app heard 'vial' as 'vile' or 'file'. It complements rather than replaces pilot data, because scripted speech differs from spontaneous speech.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Days

Steps

  1. Write a script covering key utterance types: digits, names, domain terms, commands and likely confusables.
  2. Recruit 10 to 20 speakers varied in accent, age and gender, matching the target population.
  3. Have each speaker read the script to the system in realistic acoustic conditions.
  4. Compare recognition results with the script and tally misrecognitions by item.
  5. Adjust end-of-speech timeouts and add misrecognised variants to expected phrases or grammars.
  6. Re-run problem items, then proceed to pilot testing with spontaneous speech.

Sources

  • Designing Voice User Interfaces (Pearl, 2017)

Also known as

  • Recognition testing with scripted speakers
  • Scripted recognition test

Categories

  • User Research
  • Usability & Evaluation