Description
Defining measurable acceptance criteria between stages of a research pipeline so poor-quality transcripts, parses or codes are caught before they contaminate findings.
In a multi-stage pipeline errors compound: a hallucinated transcript passage becomes a coded segment, then a theme. Quality gates set explicit, measurable thresholds at each hand-off, for example transcription WER and confidence, diarisation sanity checks, NLP component F1, LLM schema validity and quote-verification rates, and human-AI coding agreement. Failing items are halted, flagged or routed to review rather than passed silently. Rigour comes from setting thresholds before running, automating checks, recording pass and fail rates in run reports, and periodically auditing items that passed to make sure gates are not too lenient.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Latent
- Stage
- Definition, Evaluation
- Data
- Analytical / expert
- Effort
- Days
Steps
- List each stage's outputs and the failure modes that matter for findings.
- Set thresholds before running, e.g. sampled WER of 10% or below, at least 98% schema-valid outputs, 100% verbatim quote match, kappa of 0.70 or above against human coders.
- Automate checks in the pipeline and halt or quarantine failing items.
- Route quarantined items to human review with the reason for failure.
- Audit a random sample of passed items each run (e.g. 5%) to estimate leakage.
- Report gate statistics with every set of findings.
Sources
- Whisper.cpp
- spaCy
- Claude API (multi-turn agent)
- asyncio
- pandas
- Prefect/Airflow for pipeline orchestration
- Shewhart, W. A. (1931) Economic Control of Quality of Manufactured Product
- Wilson, G. et al. (2017) Good enough practices in scientific computing. PLOS Computational Biology
- The Inquiry Series, C13 — The Full Agentic Research Pipeline — Whisper → spaCy → AI Coder → Synthesis
Also known as
- Stage-gate checks
- Pipeline validation checkpoints
Categories
- Research Operations
- Research Design & Rigor