Description
A structured review of a research report that checks every causal claim against its design, flags confounds and rewrites overclaims before decisions are made.
The causal thinking audit treats a research report as data to be tested. Each sentence implying cause (drives, leads to, because of, improves) is extracted, matched to the evidence and design behind it, and checked for the threats that could produce the same pattern: confounding (such as a market rally lifting retention), selection, reverse causation, regression to the mean and collider bias. The audit then asks what counterfactual the claim implies and whether the data could distinguish it. It draws on Hernán's argument that hiding causal intent behind associational language is worse than stating it with explicit assumptions. Rigour comes from a fixed checklist, independent auditors and a documented verdict per claim.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Latent
- Stage
- Strategy, Post-launch
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Extract every explicit or implied causal claim from the report into a table.
- For each claim record the design behind it (experiment, quasi-experiment, observational log, survey, qualitative) and its rung on Pearl's ladder.
- Check each claim against a fixed threat list: confounding, selection bias, reverse causation, regression to the mean, collider bias, seasonality or market events.
- Sketch a minimal DAG for contested claims and identify any open backdoor paths.
- Assign a verdict per claim (supported, supported with stated assumptions, overclaimed, unsupported) and propose reworded language or a follow-up design.
- Have a second auditor review independently and reconcile disagreements before the report is circulated.
Sources
- Causal Inference: What If (Miguel Hernán & James Robins)
- DAGs+SEM Playbook (Vivek Gopinath)
- Nash Equilibrium Playbook (Vivek Gopinath)
- Causal Inference (Hernán & Robins)
- DAGs+SEM Playbook (Gopinath)
- Nash Equilibrium Playbook (Gopinath)
- The Inquiry Series, Ch 45 — The Causal Revolution — Pearl's Ladder & the Fundamental Problem
Also known as
- Causal claims audit
- Causal language review
Categories
- Research Operations
- Research Design & Rigor