Description
Systematically checking a sample of automated transcripts against human reference transcripts to quantify errors and decide whether they are fit for analysis.
Automated transcripts are data, and their errors propagate into coding and quotes. A transcript quality audit selects a stratified sample of recordings (covering accents, audio quality and languages), produces careful human reference transcripts for segments, and computes word error rate (WER = substitutions + deletions + insertions divided by reference words). It also checks qualitative error types that WER under-weights: hallucinated sentences during silence, mis-heard domain terms, negations dropped ('can' versus 'can't'), and mis-attributed speakers. Poland (1995) showed that even human transcription has systematic errors. Rigour comes from pre-set acceptance thresholds, stratified sampling, reporting the audit, and correcting or flagging transcripts that fail.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Latent
- Stage
- Definition
- Data
- Mixed
- Effort
- Days
Steps
- Define acceptance criteria before auditing, e.g. WER of 10% or less and zero meaning-changing errors in quoted passages.
- Draw a stratified sample of about 10% of recordings (at least 5), covering speakers, accents, languages and audio conditions.
- Create human reference transcripts for 3-5 minute segments from each sampled file.
- Compute WER with a tool such as jiwer after normalising case and punctuation.
- Review for meaning-changing errors: negations, numbers, product and drug names, hallucinated passages and speaker swaps.
- Correct failing transcripts, build a glossary or prompt for recurrent terms and re-run, and verify every quote used in reporting against audio.
Sources
- whisper.cpp (local, privacy-preserving)
- openai-whisper (cloud)
- pyannote.audio (diarisation)
- ffmpeg
- pydub
- Poland, B. D. (1995) Transcription quality as an aspect of rigor in qualitative research. Qualitative Inquiry
- Jurafsky, D. & Martin, J. H. Speech and Language Processing, 3rd ed. draft
- The Inquiry Series, C9 — Whisper — Batch Transcription at Scale
Also known as
- Word error rate audit
- ASR transcript verification
Categories
- Research Operations
- Research Design & Rigor