Measures of Effectiveness Hierarchy Across System Levels

Description

Decomposes a system into levels from workstation to command, sets suitable measures at each, and traces how operator performance and errors propagate to overall mission success.

Because human performance matters only through system output, Meister argues measurement must cover every relevant subsystem level and mission phase. Each level has its own measures: lower levels use performance measures (detection range, response time, errors), higher levels use effectiveness measures (mission success), and operator measures must be transformed as they move up (an operator's detection becomes an estimated target range for the commander). Time and error suit every level except the terminal one. The dependency of each level on the others has to be established, and measures that vary over time and across levels are the most informative. Where a direct causal trace is impossible, researchers should at least show parallel variation between human performance and system success. In product terms the levels might be screen, task, journey, account and business outcome, making this the human factors ancestor of metric trees that link UX metrics to business results.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Weeks

Steps

  1. Map the system levels and subsystems from individual workstation or screen through team and journey up to the executive or business level, noting parallel subsystems.
  2. Rank functions by importance to the system goal and select which ones to measure.
  3. Choose measures suited to each level (performance measures low, effectiveness measures high) and define how lower-level values transform into higher-level ones.
  4. Determine how each level depends on the others.
  5. Collect data on the inputs people receive, what they do with them, and how deviations such as errors and delays affect other elements.
  6. Trace, graphically if possible, the flow of effects up and down the levels to terminal output; where tracing fails, correlate variation in human performance with variation in system success.

Sources

  • Conceptual Foundations of Human Factors Measurement (Meister, 2004)

Also known as

  • MOE hierarchy
  • Tracing human performance to system success
  • Multi-level system measurement

Categories

  • Accessibility
  • Human Factors & Error