Transcription confidence scoring

Description

Using the speech model's own token probabilities and compression signals to flag low-confidence transcript segments for targeted human review.

Whisper exposes per-token and per-segment signals: average log-probability, no-speech probability and compression ratio (very repetitive output signals hallucination). Combining these yields a confidence score per segment so reviewers can focus effort where errors are most likely rather than auditing randomly. Rigour comes from calibrating thresholds against audited segments (checking that low-confidence segments really have higher WER), recognising that models can be confidently wrong, especially on domain terms, and marking flagged segments in the transcript so analysts treat quotes from them cautiously.

Attributes

Loss type
Latent
Stage
Definition
Effort
Hours

Steps

  1. Enable word or segment-level probabilities and timestamps in the transcription output.
  2. Compute per-segment signals: average log-probability, no-speech probability and compression ratio.
  3. Set initial flags using Whisper's own defaults (avg_logprob below -1.0, compression ratio above 2.4, no_speech_prob above 0.6).
  4. Calibrate by auditing flagged and unflagged samples and adjusting thresholds so flagged segments have markedly higher error rates.
  5. Mark flagged segments in transcripts and route them to human review, prioritising those in quoted or coded passages.
  6. Report the proportion of flagged content per file as a quality indicator.

Sources

  • whisper.cpp (local, privacy-preserving)
  • openai-whisper (cloud)
  • pyannote.audio (diarisation)
  • ffmpeg
  • pydub
  • Radford, A. et al. (2023) Robust speech recognition via large-scale weak supervision. ICML
  • Jiang, H. (2005) Confidence measures for speech recognition: A survey. Speech Communication
  • The Inquiry Series, C9 — Whisper — Batch Transcription at Scale

Also known as

  • ASR confidence estimation
  • Token log-probability flagging

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning