Dependency management

Description

Declaring and pinning every software library an analysis depends on, so the computational instrument can be rebuilt identically and results do not drift between runs.

In computational research the software is part of the instrument. Statistical and machine learning libraries change defaults between versions (for example a change in a clustering initialisation or a factor rotation algorithm), which can change results without any change in data. Dependency management records direct dependencies in a specification file and resolves them into a lockfile that pins every transitive package to an exact version and hash. Rigour comes from separating the human-edited specification from the machine-generated lock, updating dependencies deliberately rather than incidentally, and rerunning key analyses after any upgrade to check that results are stable. For models such as Whisper or spaCy pipelines, the model version and weights must be pinned too.

Attributes

Loss type
Unmigrated
Effort
Hours

Steps

  1. List direct dependencies with compatible ranges in pyproject.toml or environment.yml.
  2. Generate a lockfile (e.g. pip-compile, poetry lock or uv lock) that pins exact versions and hashes, and commit it.
  3. Pin model artefacts separately by name and version (e.g. en_core_web_trf 3.7.x, Whisper large-v3) and record checksums.
  4. Set random seeds and record them alongside library versions for any stochastic method (UMAP, GMM, LDA, bootstrap).
  5. Upgrade dependencies on a branch, rerun the full analysis and compare key outputs before merging.
  6. Record the full environment (pip freeze or conda env export) in the report appendix.

Sources

  • Python (pandas, numpy, pathlib)
  • Jupyter Lab
  • Git
  • conda/venv
  • cookiecutter project templates
  • Wilson, G. et al. (2017) Good enough practices in scientific computing. PLOS Computational Biology
  • The Inquiry Series, C1 — The Python Research Environment — Setup, Philosophy & Reproducibility

Also known as

  • Package pinning
  • Lockfiles
  • Environment specification

Categories

  • Research Operations
  • Research Design & Rigor