Description
Before analysing real data, generate a synthetic data set with structure you set yourself, run the planned analysis on it, and check that it recovers what you built in.
Chapman and Feit simulate data before nearly every analysis and argue the habit is valuable in itself. When you created the data, you know what the analysis should find, so you can confirm the code and model work before trusting them on real data, and you can change the simulation to see how results respond. Their examples include survey data with latent satisfaction mapped to 5-point items and non-response, segments defined by a parameter table, satisfaction drivers with a halo effect, factor and structural models from published loadings, and choice-based conjoint responses from known part-worths. The same approach doubles as design planning: varying sample size, number of questions or attributes shows the precision a budget will buy, and designs can be compared by the width of prediction intervals. Recovery shows the method works, not that the model fits reality.
Attributes
- Discipline
- Research Design & Rigor
- Loss type
- Siloed
- Stage
- Definition, Design
- Data
- Analytical / expert
- Effort
- Days
Steps
- Write down the planned analysis and the expected structure: variables, types, segments, relationships and effect sizes.
- Keep the data definition (names, distributions, segment sizes, parameter tables) separate from the generating code so it is easy to change, and set a random seed.
- Generate each variable from its distribution, building in the intended relationships, latent factors or part-worths, then convert latent scores to scale points and add floor, ceiling and non-response effects.
- Run the planned analysis on the synthetic data.
- Compare estimates with the known parameters (coefficients, loadings, segment membership, part-worths).
- Vary sample size, number of items or tasks and effect sizes, rerun, and fix the study design that gives adequate precision.
- Run the same code on the real data once collected.
Sources
- R for Marketing Research and Analytics (Chapman & Feit, 2019)
- Statistical Methods for Reliability Data (Meeker & Escobar, 1998)
- Continuous Multivariate Distributions, Vol. 1 (Kotz et al., 2000)
- A Beginner's Guide to Structural Equation Modeling (Schumacker & Lomax, 2004)
Also known as
- Simulate data with known structure, then analyse it
- Synthetic data validation of an analysis
- Parameter recovery check
- Simulation before fielding
Categories
- Research Operations
- Research Design & Rigor