Description
Records how people and a system actually perform during routine operation, with no contact or manipulation, and judges that performance against mission-derived criteria.
Meister calls UPM the simplest and most basic objective method: the researcher records what is done, how, when, where and how well, without contacting or controlling those observed, because any contact distorts performance. The researcher's control is limited to choosing when to measure, what to record and how to analyse. It cannot test causal hypotheses, but naturally occurring contrasts (night versus day, different crews, shifts) can be compared statistically. It is the method of choice when owners forbid interference, when the system is too large to manipulate, or when the aim is to describe actual performance and find errors and error-inducing features such as poor procedures or manuals. Its strength is that relevance and validity questions largely disappear because the real system is measured. It is common in acceptance tests, military exercises, medication error counts and inspection error rates, and beta testing is an instance, though such work is rarely published.
Attributes
- Discipline
- Human Factors & Error
- Loss type
- Siloed
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Learn the system thoroughly through task analyses, manuals, mission descriptions, observation of ongoing work and interviews with personnel.
- Fix the purpose and the measurement questions.
- Derive quantitative performance criteria from mission requirements; where none exist, elicit them from experts, get management agreement and link behavioural criteria to system criteria.
- Write a test plan covering subsystems, criteria, observer roles, data to collect, instrumentation, measures per subsystem, amount of data per cycle and the planned analysis.
- Pretest the procedures (by walkthrough if the system cannot run), then record performance unobtrusively during normal operation, with two observers cross-checking where needed.
- Only after the UPM period, add interviews, rating scales or experiments; then compare performance with the criteria, describe errors and natural contrasts, and trace causes of poor performance.
Sources
- Conceptual Foundations of Human Factors Measurement (Meister, 2004)
Also known as
- Unprogrammed performance measurement
- Non-interfering operational performance test
- System performance measurement in the operational environment
Categories
- Accessibility
- Human Factors & Error