Bias and Gaming Audit for Incentivised Customer-Feedback Scores

Description

Anticipate and audit four biases in customer-feedback scores (fear of retribution, bribery, sample bias, grade inflation) and validate scores against behaviour before tying them to pay.

Once feedback scores affect rankings or pay, staff have reason to manipulate them: begging for top scores, sending surveys only to happy customers, altering contact details of unhappy ones. Reichheld and Markey name four bias sources (fear of retribution, bribery, sample bias, grade inflation) and pair countermeasures with audits. Countermeasures include confidentiality (account averages in B2B), educating customers and staff, ranking teams so peers police each other, unpredictable request timing, third-party collection soon after the event, very high response rates from the right sample, imputing non-respondents and asking relative questions. Audits track failed survey contacts per unit and inspect outliers, have executives call random customers, monitor survey calls, keep methods consistent with footnoted reports, and commission external audits of lists, calls, verbatims and calculations. Scores are regularly validated against referrals, purchases and retention, and linked to pay only after several cycles of published rankings.

Attributes

Loss type
Latent
Stage
Post-launch
Data
Mixed
Effort
Weeks

Steps

  1. List how each of the four bias sources (retribution, bribery, sample bias, grade inflation) could arise in your setting.
  2. Choose countermeasures for each, such as third-party collection, unpredictable timing and confidentiality.
  3. Track contact-failure and response rates by unit and investigate outliers.
  4. Spot-check by calling random customers and detractors and monitoring survey calls.
  5. Keep the collection method consistent and document it in report footnotes; consider an external audit.
  6. Validate scores against later customer behaviour (referrals, purchases, retention).
  7. Delay linking scores to pay until they have proved stable and valid over several cycles.

Sources

  • The Ultimate Question 2.0 (Reichheld & Markey, 2011)

Also known as

  • Feedback score integrity audit
  • Speeding (Enterprise)

Categories

  • Research Operations
  • Research Design & Rigor