Description
Fixed roles and ground rules that let stakeholders watch test sessions live without disturbing participants, capture their top problems and feed vetted questions to the moderator.
Watching even one user live changes stakeholders more than any report, so the protocol aims to get influential people in front of sessions while protecting data quality. Each session has a moderator and a note-taker, rotating between sessions, and no more than two researchers in the room. Observers watch from a separate room or remotely, are briefed on rules (silence, phones off, write questions down, never interrupt, expect task order to vary), and anyone listening in remote sessions is introduced. Managers of participants are excluded, because people then describe policy rather than practice. After every session each observer writes the three most serious problems they saw. A 'hall monitor' manages the observation room and relays questions; near the end the moderator collects written questions, filters out leading ones and asks the useful ones in a closing 'coffee talk'.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Active
- Stage
- Evaluation
- Data
- Qualitative
- Effort
- Hours
Steps
- Schedule sessions on a quiet day and invite everyone with influence on the design, with a save-the-date about two weeks ahead and reminders.
- Assign a moderator and note-taker per session, rotate roles, and limit the test room to two researchers.
- Set up an observation room or remote feed out of earshot, with a shared screen and good audio; record for absentees.
- Brief observers: silence, phones off, write questions down, never interrupt, note quotes with timestamps; introduce all remote listeners to the participant and exclude participants' managers.
- After each session, have every observer write their three most serious problems on a form.
- Near the end of each session, collect observer questions, filter and reword leading ones, and ask them in a closing segment.
- Have all observers attend the debrief with their lists.
Sources
- Rocket Surgery Made Easy (Krug, 2010)
- UX Research (Nunnally & Farkas, 2017)
- Understanding Your Users (Baxter et al., 2015)
- Handbook of Usability Testing (Rubin & Chisnell, 2008)
Also known as
- Team observation of test sessions
- Make it a spectator sport
- Stakeholder observation of usability tests
- Stakeholder observation of live sessions
- Observation room protocol
- Observer ground rules
- Moderator and note-taker roles
- Hall monitor
- Coffee talk
Categories
- User Research
- Usability & Evaluation