Description
Gives each respondent a weight so the sample's make-up on known characteristics matches the real user population before results are reported.
Survey weighting corrects a sample that is known to be unbalanced. Veal and Darcy's example is a site survey where interviews were spread evenly through the day, while counts showed that more than half of users came around midday; because midday users may differ, they are under-represented. For each group, such as a time-period, the weighting factor is the number in the population divided by the number in the sample, so each interview 'grosses up' to the users it represents. The weighted sample then matches the population. They note that the true sample size has not changed, so for significance tests the factors should be rescaled so the weighted total equals the actual number of interviews. Any characteristic known for the population can be used, such as age from the census, membership records or observed gender balance. They also suggest weighting to correct for different response rates in mail surveys. For UX surveys, which often over-represent engaged or recent users, weighting against analytics data on plan, tenure, platform or activity is a cheap way to make results more representative.
Attributes
- Discipline
- Statistics & Modelling
- Loss type
- Siloed
- Stage
- Evaluation, Post-launch
- Data
- Quantitative
- Effort
- Hours
Steps
- Identify a characteristic that matters for the results and is known for the whole user population (for example time of use, plan, platform, age group).
- Tabulate the sample and the population across the categories of that characteristic.
- Compute a weighting factor for each category as population number divided by sample number.
- Rescale the factors so the weighted total equals the actual sample size if significance tests will be run.
- Attach the weights as a variable and apply them in the analysis software.
- Compare weighted and unweighted results and report both the weighting basis and any large differences.
- Avoid weighting on characteristics with very small sample cells, or combine categories first.
Sources
- Veal, A. J. & Darcy, S., Research Methods in Sport Studies and Sport Management (2014)
Also known as
- Post-stratification weighting
- Weighting to correct sample bias
- Grossing up
- Cell weighting
- Raking
- Iterative proportional fitting
- Rim weighting
Categories
- Analytics & Behaviour
- Statistics & Modelling