Description
A hidden human operator simulates system intelligence such as speech, AI or recommendation logic, letting you test interaction concepts before the technology exists.
In a Wizard of Oz study the participant interacts with what appears to be a working system while a hidden 'wizard' produces the system's responses in real time. It lets teams study how people speak to, trust and recover from errors with intelligent systems before investing in models or infrastructure. Rigour depends on wizard consistency: a written response protocol, constrained response libraries, deliberate simulated error rates, and timing that mirrors realistic latency. Without these, the wizard behaves like an idealised human, inflating satisfaction and hiding the failure modes real systems produce. Ethical practice requires debriefing participants about the deception. The method is widely used for conversational, voice and generative AI concepts, and for exploring how much automation people want.
Attributes
- Discipline
- Usability & Evaluation
- Loss type
- Active
- Stage
- Design
- Data
- Qualitative
- Effort
- Days
Steps
- Define the system capabilities to simulate and the questions to answer (vocabulary, turn-taking, trust, error recovery).
- Write a wizard protocol with response rules, a library of permitted replies, target latency and a planned error rate (for example 10-20% misrecognitions).
- Build a front end that routes participant input to the wizard and wizard output back to the participant without revealing a human.
- Pilot with 2-3 people to train the wizard and check the illusion holds and responses stay consistent.
- Run 6-12 sessions, logging all inputs, wizard responses, timings and participant reactions, with a separate observer.
- Debrief participants about the simulation, then analyse utterance patterns, failure recovery and expectations to specify the real system.
Sources
- Universal Methods of Design
- Chatbot Research and Design
- Kelley, An iterative design methodology for user-friendly natural language office information applications (ACM TOIS, 1984)
- Lost Methods in UX — A Complete Treatise (Gopinath)
- Kelley JF (1984) An iterative design methodology for user-friendly natural language office information applications. ACM TOIS 2(1)
- Dahlbäck N, Jönsson A & Ahrenberg L (1993) Wizard of Oz studies: why and how. IUI
- Gould JD, Conti J & Hovanyecz T (1983) Composing letters with a simulated listening typewriter. CACM 26(4)
Also known as
- Wizard of Oz testing
- WoZ
- Human-in-the-loop simulation
- Wizard of Oz simulation
- Wizard of Oz Prototyping (Full Method)
Categories
- User Research
- Usability & Evaluation