Description
Usability testing adapted to speech interfaces: observation instead of think-aloud, concealed speech capability, and voice-specific measures of error type, recovery and completion.
Concurrent think-aloud is impossible while a participant is talking to a system, so voice testing relies on observation, recordings, facial expression and post-task questions. Test a working build after a couple of pilot participants have caught gross recognition problems, since the aim is flow, not recognition accuracy. Do not reveal that the app accepts speech unless users would know in real life; whether and when users realise they can speak is a finding. Define task completion per task in advance and also ask users whether they achieved their goal, because users may fail without minding. Count errors by type (rejection versus mismatch) and record whether and how quickly users recovered. For cars, test safely in a mock-up rig with a simulator, in a parked car, or with a back-seat Wizard playing prompts, while watching cognitive, visual and manual distraction.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Siloed
- Stage
- Design, Evaluation
- Data
- Mixed
- Effort
- Weeks
Steps
- Prepare a working build or Wizard of Oz set-up with fake accounts or hard-coded data, and pilot with one or two participants to fix gross recognition problems.
- Write goal-based tasks without command words and rotate their order.
- Do not tell participants the app accepts speech unless they would know in real life.
- Observe without think-aloud: speaking cues, exact utterances, hesitation, duration, expressions; step in only for crashes or high frustration.
- Ask brief questions after each task and a questionnaire at the end; code completion against predefined criteria and ask whether users achieved their goal.
- Count errors by type (rejection, mismatch) and recovery success and time.
- For in-car use, test in a simulator rig, a parked car or with a back-seat Wizard, and record distraction; then tally, rank issues by severity and share a fix plan.
Sources
- Designing Voice User Interfaces (Pearl, 2017)
Also known as
- VUI usability testing
- Speech interface usability test
- In-vehicle voice interface testing
- Parked-car testing
- Automotive VUI Wizard of Oz
Categories
- User Research
- Usability & Evaluation