ASPIRE Risk-Weighted Quality Review of Recurring Surveys

Description

A structured review that rates each error source of a recurring survey on five criteria, weights it by risk and produces a quality score tracked across cycles.

ASPIRE operationalises the total survey error paradigm as a repeatable audit for survey products that run every cycle. Total error is decomposed into sources (sampling, frame, nonresponse, measurement, data processing, modelling and estimation). Reviewers gather evidence from documentation, self-assessments and a quality interview with programme staff and users, then rate each source on five criteria: knowledge of the risk, communication with providers and users, compliance with best practice, available expertise, and progress on mitigation plans. Ratings use defined levels from poor to excellent, are weighted by each source's intrinsic risk (low, medium, high) and roll up into source and product scores. Product owners comment before scores are fixed, and the review repeats so improvement can be tracked. Its weakness is subjectivity: scores depend on raters and do not capture all of total survey error.

Attributes

Loss type
Unmigrated
Effort
Weeks

Steps

  1. List the error sources relevant to the recurring survey (sampling, frame, nonresponse, measurement, processing, estimation).
  2. Gather evidence: documentation, quality declarations, self-assessments and a structured interview with the survey owners and main users.
  3. Rate each source on five criteria (knowledge of risk, communication, compliance with best practice, expertise, achievement of mitigation plans) using defined levels from poor to excellent.
  4. Assign each source an intrinsic risk weight (low, medium, high) and compute source and overall product scores.
  5. Let product owners review and comment before finalising.
  6. Produce a prioritised improvement list and repeat the review each cycle to track change.

Sources

  • The SAGE Handbook of Survey Methodology (Wolf et al., 2016)

Also known as

  • ASPIRE
  • A System for Product Improvement, Review and Evaluation
  • Statistics Sweden quality review
  • Risk-weighted error source rating

Categories

  • Research Operations
  • Research Design & Rigor