Controlled Experiments with Groups (Groupware Evaluation)

Description

Plan controlled evaluations of collaborative systems around tasks that force cooperation, with per-participant recording and analysis strategies that survive large group-to-group variation.

Evaluating a collaborative system means the unit of analysis is the group, not the person, which multiplies participant numbers and session length so only a few groups can be run. The method is a set of design rules for such experiments. Tasks must require consensus or distributed information and control, and must use the channels under test: shared report writing, design on a shared pinboard, ranking games such as desert survival, or time-critical simulated process control. Each participant's view, sound, keystrokes and video are recorded so their experience can be relived, with only rough synchronisation across people. Because groups vary so much, even a clean result needs at least four groups per condition to reach 5 percent significance, so analysts use within-group designs, micro-analysis of measures such as gaps between utterances, or incident analysis of breakdowns. Unlike group task analysis, which models the work, this compares system conditions experimentally.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Weeks

Steps

  1. Estimate participant numbers in groups rather than individuals, and allow session time for the group to build rapport.
  2. Choose a task that requires consensus or distributed information and control, and that exercises the communication channels the system provides.
  3. Record, for each participant, what they saw and heard plus their own input and video, synchronising roughly across participants.
  4. Decide the analysis strategy in advance: within-group comparison of conditions, micro-analysis of fine-grained measures (for example gaps between utterances) suited to parametric tests, or critical-incident analysis of breakdowns using conversation analysis.
  5. Plan for at least four groups per condition if a between-groups significance test is intended, or switch to a within-group design.
  6. Spend groups on conditions likely to be informative rather than strict controls, and consider a field study of an existing team where group formation would distort the result.

Sources

  • Human-Computer Interaction (Dix et al., 2004)

Also known as

  • Experiments with groups of users
  • Group-working experiments
  • Controlled evaluation of groupware

Categories

  • User Research
  • Usability & Evaluation