Description
Planning sample sizes with design effects for weighting and clustering, then analysing data with strata, clusters and weights declared so standard errors are correct.
Clustering and unequal weights make a sample less informative than its nominal size. The design effect (Deff) quantifies this: the weighting component Deffp comes from variation in selection probabilities, and the clustering component is Deffc = 1 + (b - 1) x rho, where b is mean cluster size and rho the intraclass correlation. Their product gives the overall Deff, and the nominal sample needed is the effective sample size multiplied by Deff. The European Social Survey plans equal effective sample sizes across countries this way, with Deffp typically 1.20 to 1.25. At analysis, the design is declared to software (strata, clusters, weights) and standard errors come from Taylor linearisation or replicate weights (jackknife, balanced repeated replication, bootstrap). Subgroups are analysed with subpopulation options rather than by deleting cases. Ignoring the design understates standard errors and overstates significance.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Design, Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Estimate the weighting design effect from expected variation in selection probabilities or weights.
- Estimate the clustering design effect as 1 + (b - 1) x rho from mean cluster size b and an intraclass correlation rho from prior data.
- Multiply to get the overall Deff and set the nominal sample size as effective sample size times Deff.
- Prefer more clusters (for example more accounts or sites) over larger clusters when budgets allow.
- At analysis, declare strata, clusters and weights in survey software (R survey, Stata svy).
- Estimate standard errors with Taylor linearisation or replicate weights.
- Analyse subgroups with subpopulation options instead of subsetting the data.
Sources
- The SAGE Handbook of Survey Methodology (Wolf et al., 2016)
- Quantitative Social Research Methods (Singh, 2007)
- Beyond t Test and ANOVA (Yu et al., 2022)
- Applied Logistic Regression (Hosmer & Lemeshow, 2000)
Also known as
- Design effect
- Deff
- Taylor linearisation
- Replicate weights (JRR, BRR, bootstrap)
- Survey-weighted analysis
Categories
- Analytics & Behaviour
- Statistics & Modelling