Report-first survey design with dummy tables

Description

Draft the report, placeholder conclusions and every table before writing questions, and confirm what each possible result would change, so every question and the sample size earn their place.

Designing a study backwards from its deliverable prevents 'just in case' questions and research no one will act on. Goodman et al. describe drafting the report's opening as if the survey were finished (purpose, goals, design, population, sampling, sample size, completion rate, analysis), writing hoped-for conclusions as placeholder hypotheses, listing the data each needs, and building every table and graph with its variables labelled; questions are then derived from the tables and pilot data are run through the dummy tables. Chapman and Rodden add the stakeholder side: present hypothetical results ('suppose CSat drops 2 points; what if 5?') and ask what action each would trigger. If nothing would change, clarify the decision first; if only a 5-point change matters, size the study to detect it. A full form simulates data and drafts the whole report. Ask for the latest date the answer is still useful.

Attributes

Loss type
Siloed
Effort
Hours

Steps

  1. Identify the decision the stakeholder faces and ask 'what is the latest date this is still useful?'.
  2. Present hypothetical results of different sizes and directions and record what action each would trigger; if none, clarify the decision before researching.
  3. Draft the report's opening as if finished: purpose, goals, design, population, sampling method, sample size, expected completion rate and analysis.
  4. Write the hoped-for conclusions as placeholder hypotheses and list the data each requires, including subgroup sizes.
  5. Build every table and chart as a dummy, labelled with which variables are plotted against which; optionally fill with simulated data.
  6. Derive the questions from the tables, dropping any that feed no table, and set the sample size from the smallest effect that would change the decision.
  7. After pretesting, run pilot data through the dummy tables to check the analysis works, and open the final report with the key questions and short answers.

Sources

  • Observing the User Experience (Goodman et al., 2012)
  • Quantitative User Experience Research (Chapman & Rodden, 2023)
  • Survey Design (Sauro & Lewis, 2021)
  • Usability Engineering (Nielsen, 1993)
  • Statistics for Marketing and Consumer Research (Mazzocchi, 2008)
  • Customer Analytics for Dummies (Sauro, 2015)

Also known as

  • Dummy tables
  • Report-first questionnaire design
  • Working backward from hypothetical results
  • Work backward
  • Mock deliverable review
  • Decision-threshold elicitation
  • Research grid

Categories

  • Analytics & Behaviour
  • Survey & Psychometrics