AI Design Complexity Mapping (Four Levels of AI Systems)

Description

Place an AI system on two axes, capability uncertainty and output complexity, to find its level and anticipate which design challenges and prototyping methods will apply.

Yang traces the difficulty of designing with AI to two sources. Capability uncertainty is how unclear it is what the system can do and how well: high before a dataset is fixed, narrower once a model is trained, and wide again when the system keeps learning after launch and performs differently for different users and contexts. Output complexity runs from a few fixed outputs whose errors can be listed, through many outputs a designer can still simulate, to open-ended outputs that neither rules nor a wizard can mimic. Crossing the two gives four levels. Level 1 systems, such as a classifier with fixed classes trained on a fixed dataset, can be designed with familiar methods; Level 4 systems, which learn after deployment and produce open-ended output, strain understanding of capability, ideation, prototyping and collaboration with engineers. The map helps a team choose methods and anticipate risks early.

Attributes

Loss type
Siloed
Effort
Hours

Steps

  1. Describe the system's capability: does it learn from a fixed dataset or keep learning after deployment, and does performance vary by user or context?
  2. Describe its outputs: a few fixed classes, many outputs that can be simulated, or open-ended outputs that resist simulation.
  3. Place the system on the map and agree its level from 1 to 4 with designers and engineers together.
  4. For Level 1, explore the model directly, list true and false positive and negative cases, and design each case with ordinary wireframes.
  5. For higher levels, plan closer designer-engineer collaboration, prototyping that simulates evolving or open-ended behaviour (for example Wizard of Oz Prototyping with realistic errors) and evaluation after launch.
  6. Optionally map a specific concern, such as fairness, along the path from algorithm to deployed system to lived experience to see which parts are being neglected.

Sources

  • Profiling Artificial Intelligence as a Material for User Experience Design (Yang, 2020)

Also known as

  • AI design complexity framework
  • Capability uncertainty x output complexity map
  • Yang's four levels of AI systems

Categories

  • Design
  • Design & Product Practice