Out-of-Box Setup Testing

Description

Observe people unboxing and setting up a physical product unaided, revise guides and packaging, and retest weekly until a simple time or complaint threshold is met.

Connected and consumer hardware products often fail at first contact: people skim quick-start guides, misread status lights and are unsure of cable orientation. Out-of-box testing hands participants the product exactly as shipped and watches unboxing, guide use, physical connection and registration without help. Its distinctive rigour is a simple stopping rule. BRCK set a 10 to 20 minute set-up target; Sifteo, unable to change a battery mechanism, tested ten arrow-sticker versions and each week had 10 to 15 people try set-up until fewer than 2 in 10 complained. Because problems are often found after hardware is frozen, fixes target the cheapest layers first: instructions, labels, packaging, videos and online help. Support contacts after launch continue the evaluation.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Weeks

Steps

  1. Give participants the packaged product exactly as shipped, with no briefing beyond the scenario.
  2. Observe unboxing, use of printed guides, physical connection, power-up and registration without help.
  3. Record confusions and time to a working state.
  4. Set a simple threshold, such as set-up within 10 to 20 minutes or fewer than 2 in 10 participants complaining.
  5. Revise the cheapest layers first when hardware is frozen: instructions, labels, stickers, packaging, videos and online help, keeping print and on-screen diagrams identical to the shipped part.
  6. Retest weekly with 10 to 15 fresh participants until the threshold is met.
  7. After launch, keep monitoring set-up problems through support contacts.

Sources

  • Designing Connected Products (Rowland et al., 2015)

Also known as

  • Out-of-Box Setup Testing with a Complaint Threshold
  • Unboxing usability test
  • First-run setup testing
  • Out-of-box experience testing

Categories

  • User Research
  • Usability & Evaluation