Emotion detection from mouse and keyboard logs

Description

Train classifiers to infer users' emotions from mouse and keyboard features, using webcam-based labels during training, so emotion can later be estimated without a camera.

Desolda and colleagues collected in-the-wild data with a browser plug-in that logged mouse and keyboard events during ordinary web use for weeks and captured a webcam face image every two seconds. Facial expression analysis provided emotion labels. Features derived from interaction logs were used to train classifiers (decision trees, random forests, AdaBoost and a multilayer perceptron), with forward feature selection per model. Once trained, the model can run in a page script to infer visitors' emotions in real time from interaction alone. With 12 participants over 30 days, the best average accuracy was 47 per cent (random forest), so the approach remains exploratory. Labels inherit the error of face-based emotion recognition, test reliability cannot be tracked, and continuous webcam capture needs strong consent and data protection.

Attributes

Loss type
Latent
Effort
Months

Steps

  1. Recruit volunteers and obtain explicit consent for interaction logging and periodic webcam capture.
  2. Have participants install a logging plug-in and use the web normally for several weeks.
  3. Label emotional state from face images at short intervals (every 2 seconds in the study).
  4. Derive features from interaction logs (speed, pauses, clicks, typing rhythm) aligned with labels.
  5. Train and compare classifiers with forward feature selection, validating on held-out participants rather than held-out samples.
  6. Deploy the best model only if accuracy is well above chance, and validate its signal against self-report before use.

Sources

  • User Experience Design Using Machine Learning (Abbas et al., 2022)

Also known as

  • Emotion detection from mouse and keyboard interaction logs
  • Interaction-log affect detection
  • Mouse and keyboard emotion recognition
  • Non-intrusive affective modelling

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning