Automated facial expression analysis

Description

Uses computer vision to detect facial action units and estimated expressions from video during product use, providing time-stamped indicators of affective response.

FACS decomposes facial movements into action units, such as AU4 brow lowerer and AU12 lip corner puller, which human coders score after about 100 hours of training. Automated systems detect action units and often infer emotion categories from video at frame level. In UX this yields continuous, time-aligned signals during tasks without interrupting participants. However, Barrett and colleagues' 2019 review concluded that facial movements do not reliably map to specific emotions across people and contexts; confusion often shows as AU4, and many users show little expression while using screens. Rigour therefore means using action units as signals rather than emotion labels, controlling lighting and camera angle, using within-person baselines, and triangulating with self-report and behaviour.

Attributes

Loss type
Siloed
Stage
Evaluation
Effort
Weeks

Steps

  1. Set up recording with a frontal camera at face height, even lighting and at least 720p at 30 frames per second, and check detection quality in a pilot with 2 to 3 people of varied skin tones, ages and with or without glasses.
  2. Record a neutral baseline of 1 to 2 minutes per participant, then record task sessions with time stamps for task events.
  3. Run automated detection and export action unit intensities per frame, excluding frames with poor tracking confidence (for example below 0.8).
  4. Compute within-person changes from baseline around task events, focusing on interpretable signals such as AU4 (effort or confusion) and AU12 (smiling).
  5. Validate a sample of automated detections against manual FACS coding or careful human review, and report agreement.
  6. Triangulate facial signals with self-report and behaviour, and report them as indicators of response at specific moments rather than definitive emotions.

Sources

  • Unmasking the Face (Ekman & Friesen)
  • Designing Across Senses (Noessel)
  • Affective Computing (Picard)
  • The Inquiry Series, Ch 38 — FACS, Micro-Expression Analysis & Psychophysiology

Also known as

  • Facial coding
  • Automated FACS
  • Facial action unit detection
  • Emotion AI facial analysis

Categories

  • Analytics & Behaviour
  • Neuro, Biometrics & Affect