Description
Checks each tracked metric against explicit quality criteria and known failure signs, dropping vanity metrics that change no decision and fixing those that invite gaming.
The audit combines two checklists. Croll and Yoskovitz define a good metric as comparative, understandable, a ratio or rate, and behaviour-changing; the test for vanity is to ask what you will do differently because of the number. Totals such as sign-ups, hits, page views, followers and downloads are vanity; percentage of users active is actionable. They also classify metrics as qualitative or quantitative, exploratory or reporting, leading or lagging, correlated or causal. Frost adds seven signs of success (pedigree back to a critical success factor, line of sight to action, proper comparatives, encourages right actions, valid and reliable, graphic reporting, cost effective) and eight signs of trouble, including one-number indices that mask offsetting changes, starting from available data (the lamp-post problem), ratios that invite denominator management, unquestioned vendor metrics, complex formulas and drowning in metrics.
Attributes
- Discipline
- Product Analytics & Strategy
- Loss type
- Vestigial
- Stage
- Strategy, Post-launch
- Data
- Analytical / expert
- Effort
- Hours
Steps
- List the metrics the team checks regularly, starting with the three to five looked at daily.
- For each, check whether it is comparative, understandable, a rate or ratio, and traceable to a goal or critical success factor.
- Ask what you would do differently based on it; mark those without an answer as vanity.
- Check for signs of trouble: composite indices, metrics chosen because data existed, ratios whose denominator can be gamed, vendor defaults, formulas outsiders cannot explain, and too many metrics.
- Test comprehension by asking people outside the team to explain each metric.
- Drop or replace failing metrics and pair related ones (for example conversion rate with time to purchase) to expose trade-offs.
Sources
- Lean Analytics (Croll & Yoskovitz, 2013)
- Designing Metrics (Frost, 2007)
Also known as
- Vanity metrics audit
- Actionable metrics check
- Metric quality criteria
- Signs of success and signs of trouble checklist
- Denominator management
- Lamp-post problem
Categories
- Analytics & Behaviour
- Product Analytics & Strategy