Parallel processing with async

Description

Running many independent analysis tasks concurrently, such as LLM coding requests per transcript segment, to cut processing time while respecting API rate limits.

Most AI-assisted coding time is spent waiting for API responses. Python's asyncio lets a pipeline send many requests concurrently, bounded by a semaphore so rate limits and costs are respected, and gather results as they complete. This turns days of sequential processing into hours. Rigour concerns are practical but real: concurrency limits, exponential backoff on rate-limit errors, idempotent writes so partial runs resume without duplication, deterministic ordering of results, and logging per request. Concurrency must not change the analysis: each segment should get identical prompts and settings regardless of order.

Attributes

Loss type
Unmigrated
Effort
Days

Steps

  1. Identify tasks that are independent (e.g. coding each segment) and suitable for concurrency.
  2. Use an async client and an asyncio.Semaphore to cap concurrent requests within provider limits (e.g. 5-20).
  3. Implement retries with exponential backoff and jitter for rate-limit and transient errors.
  4. Write each result keyed by segment ID so reruns skip completed segments.
  5. Record per-request metadata (model, latency, tokens, errors) and monitor cost.
  6. Verify after the run that every input has exactly one valid output.

Sources

  • Whisper.cpp
  • spaCy
  • Claude API (multi-turn agent)
  • asyncio
  • pandas
  • Prefect/Airflow for pipeline orchestration
  • Python Software Foundation. asyncio documentation
  • The Inquiry Series, C13 — The Full Agentic Research Pipeline — Whisper → spaCy → AI Coder → Synthesis

Also known as

  • Asynchronous API calls
  • Concurrent processing

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning