Multi-stream field capture (computational ethnography)

Description

A portable sensor kit records audio, gaze, body position, room and screen video and input events in a real setting, then aligns the streams for joint analysis.

Developed to study how physicians use electronic medical records during patient visits, the approach packs a microphone array, remote eye tracker, webcam, depth camera, room video and screen and input capture into a rolling case that can be set up in about ten minutes. Analysis synchronises all streams, segments body-tracking data into clinician, patient and objects, derives gaze and gestures, separates speakers by audio direction, analyses computer events, and displays everything on a shared timeline. The rigour lies in making observation of a real, unrepeatable encounter quantifiable and replayable: questions such as how long a clinician looks at the screen while the patient speaks can be answered exactly rather than estimated from field notes.

Attributes

Loss type
Siloed
Data
Mixed
Effort
Weeks

Steps

  1. Assemble the sensors (microphone array, eye tracker, cameras, depth sensor, screen and input logger) in a portable case.
  2. Agree consent and privacy procedures with everyone who will be recorded, including bystanders.
  3. Install the kit around the workstation in the real room; aim for a set-up of about ten minutes so routine work is not disrupted.
  4. Record complete encounters rather than fragments.
  5. Synchronise audio and video streams to a common clock.
  6. Segment body-tracking data into people and objects, derive gaze and gestures, and separate speakers using directional audio.
  7. Analyse computer events and screen capture, then review all streams together in a timeline tool to code episodes.

Sources

  • Research Methods in HCI (Lazar et al., 2017)

Also known as

  • LAB-IN-A-BOX
  • Computational ethnography
  • LAB-IN-A-BOX field multi-stream capture
  • Multimodal field recording

Categories

  • User Research
  • Field & Ethnographic