Description
Segment recorded speech video into utterances, extract text, voice and facial features for each, fuse them, and classify each utterance's sentiment or emotion.
Spoken opinions carry meaning in tone and face as well as words, so combining channels can classify affect better than text alone. Each video is cut into utterances, which annotators label for polarity or emotion. Text features come from the transcript (neural features, concept vectors, sentiment lexicons, negation flags); visual features from tracked facial landmarks and head pose, summarised per utterance; audio features from a toolkit such as openSMILE (pitch, intensity, spectral and voice-quality measures). Features are fused at feature level (concatenated and fed to one classifier, for example multiple kernel learning), at decision level (separate classifiers whose probabilities are combined) or in hybrids. On a Spanish product-review set the book reports about 88% accuracy for three modalities against 79% for text alone, but gains are much smaller on natural than acted data.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Siloed
- Stage
- Evaluation, Post-launch
- Data
- Mixed
- Effort
- Weeks
Steps
- Collect speech videos, preferably natural rather than acted, with consent for automated analysis, and segment each into utterances.
- Have annotators label each utterance's sentiment or emotion and check inter-annotator agreement.
- Transcribe and extract text features (embeddings, lexicon polarity, negation and intensifier flags).
- Track facial landmarks and head pose per frame and summarise features by mean and SD over each utterance.
- Extract audio descriptors over short windows after per-speaker normalisation and summarise them per utterance.
- Fuse by concatenation with one classifier, by combining per-modality classifier probabilities, or by a hybrid; detect and down-weight missing or noisy channels.
- Evaluate with speaker-independent cross-validation and per-class confusion matrices, comparing unimodal and fused results.
Sources
- Multimodal Sentiment Analysis (Poria et al., 2018)
Also known as
- Multimodal sentiment analysis
- Tri-modal sentiment analysis
- Multimodal affect recognition
- Audio-visual-textual emotion recognition
- Multimodal fusion for affect detection
Categories
- Analytics & Behaviour
- Computational & Machine Learning