Prompt UX Research

Description

Research on prompt-driven interfaces that tests how people discover capabilities, form mental models of the AI, phrase requests and recover from poor outputs.

Prompt UX research addresses the 'blank box' problem of generative and conversational interfaces: users must guess what the system can do and how to ask. It studies capability discoverability, the mental models people form about the model's knowledge and limits, prompt formulation strategies, iteration and repair after unsatisfactory outputs, and trust calibration. Typical designs combine task-based sessions with think-aloud, analysis of logged prompt sequences (reformulation rate, turns to success, abandonment), and mental-model elicitation before and after use. Rigour comes from realistic open tasks rather than scripted prompts, coding prompt logs against an intent and repair taxonomy, and distinguishing failures of the model from failures of the interface such as missing suggestions, poor affordances or unclear scope.

Attributes

Loss type
Active
Data
Mixed
Effort
Days

Steps

  1. Define the core user intents and the capability boundaries you want users to discover.
  2. Elicit prior mental models: ask 6-12 participants what they expect the assistant can and cannot do before first use.
  3. Run open-ended task sessions with think-aloud, letting participants phrase requests in their own words.
  4. Log every prompt, response and reformulation, and code them for intent, strategy, repair type and outcome.
  5. Compute behavioural indicators from logs at scale: reformulation rate, turns to success, abandonment after first response, use of suggested prompts.
  6. Re-elicit mental models after use, compare with actual capabilities, and identify interface changes such as starter prompts, scoping cues or examples.

Sources

  • AI for Humans
  • Chatbot Research and Design
  • Designing Voice User Interfaces (Cathy Pearl)
  • Amershi et al., Guidelines for Human-AI Interaction (CHI 2019)

Also known as

  • Prompt interface research
  • Conversational AI usability research
  • Prompt discoverability testing

Categories

  • User Research
  • Usability & Evaluation