External and Multi-Instrument Validation of Analytics Metrics

Description

Treats analytics figures as estimates and bounds their error by comparing independent instruments, such as two analytics installs or panel data, before relying on them.

Most numbers about people at scale are measurements, not counts: they come from a black-box chain of collection, filtering, sampling and modification, yet dashboards show them as crisp figures that anchor decisions. Jansen and colleagues show that two independently configured analytics installations report different numbers for the same site, panel-based and site-centric tools disagree, and sentiment tools disagree on the same texts. The method compares two or more independent instruments on the same phenomenon to establish a discrepancy range from which error can be estimated, and reports figures as estimates with that range. Kohavi et al. add comparison with external sources such as panels, surveys and published data, judged by agreement in trend and seasonality rather than absolute level. Agreement does not prove accuracy, but disagreement reveals fragile metrics.

Attributes

Loss type
Latent
Effort
Weeks

Steps

  1. List the key numbers a decision rests on.
  2. Identify at least two independent instruments for each, for example site-centric and panel-based analytics, two analytics installs, two sentiment tools or two human labellers.
  3. Align definitions, scope and time periods so the instruments measure the same thing.
  4. Collect the metric from each instrument and compute the discrepancy range.
  5. Compare time series for agreement in trend and seasonality rather than absolute levels.
  6. Report figures as estimates with their discrepancy range, and investigate large disagreements rather than settling by majority.

Sources

  • Trustworthy Online Controlled Experiments (Kohavi et al., 2020)
  • Understanding Audiences via Analytics (Jansen et al., 2024)

Also known as

  • External data validation of internal metrics
  • Benchmarking against panel data
  • Multi-instrument discrepancy check
  • Counting versus measuring check
  • Analytics data validity check
  • Dual-install analytics comparison

Categories

  • Analytics & Behaviour
  • Product Analytics & Strategy