Usability Testing of Interface Standards

Description

Test whether designers can understand and apply an interface standard by having them design a toy interface with it and list the seeded violations in a sample interface.

Nielsen argues that an in-house interface standard is itself a product whose users are developers, and that it can have 'meta-usability' problems. Thovtrup and Nielsen used two tasks: designers build a small toy interface using the standard, and they inspect a small interface seeded with known violations and list them. Observation, optionally with thinking aloud, shows how they use the document. Toy designs are scored for conformance and violation lists are compared with an answer key to count misses and false alarms. The evidence for doing this is sobering: Mosier and Smith found 58 percent of guideline users found what they sought, designers using a draft ISO menu standard violated 11 percent of its rules, and three real products broke 7 to 12 rules of their own company standard. Developers rely on examples and checklists more than body text, so examples must be fully compliant.

Attributes

Loss type
Extinct
Data
Mixed
Effort
Days

Steps

  1. Recruit a few designers or developers who will actually use the standard.
  2. Task 1: ask them to design a small toy interface using the standard, observing their use of the document, optionally with think-aloud.
  3. Score the toy designs for conformance with each rule.
  4. Task 2: give them a small interface seeded with a known number of violations and ask them to list every violation.
  5. Compare their lists with the answer key, counting missed violations and legal elements wrongly flagged.
  6. Revise the standard, especially its examples and checklists, and retest with new designers.

Sources

  • Usability Engineering (Nielsen, 1993)

Also known as

  • Meta-usability testing of style guides
  • Standards usability test
  • Testing a design standard with designers

Categories

  • User Research
  • Usability & Evaluation