Description
Before a draft metric set is deployed, a group reviews each metric to forecast the behaviours and consequences it will provoke, and revises the set accordingly.
Metrics change behaviour, often in ways nobody intended. Frost presents predictive forecasting as the last of five things to get right when starting a metrics design: once a set of metrics is drafted, a group of observers works through it one metric at a time, like organised brainstorming, asking what will change once it is deployed, what new behaviours people may show, which existing good and bad behaviours may change, and what further consequences may follow. He advises including fresh eyes and a cynical thinker or two, and reports that the exercise usually produces surprising insights and important revisions; teams that skip it meet the surprises in live use. It complements his 'unintended consequences' sign of trouble and a design, use, audit and improve cycle.
Attributes
- Discipline
- Product Analytics & Strategy
- Loss type
- Latent
- Stage
- Strategy, Definition
- Data
- Analytical / expert
- Effort
- Hours
Steps
- Draft the metric set and its intended use (targets, incentives, dashboards).
- Assemble observers including people outside the team and at least one sceptic.
- For each metric, ask what will change once it is deployed and reported.
- Ask what new behaviours may appear and which existing good or bad behaviours may change, including gaming of numerators or denominators.
- Trace further consequences of those changes for users, staff and other metrics.
- Revise metrics or add balancing and guardrail metrics before implementation, and schedule a post-launch audit.
Sources
- Designing Metrics (Frost, 2007)
Also known as
- Predictive Forecasting
- Pre-deployment metric consequence review
Categories
- Analytics & Behaviour
- Product Analytics & Strategy