Cross-Tool Log Sessionisation with Diary Validation

Description

Joins logs from many tools into task sessions, derives time-spent metrics, and validates them against participant diaries before using them to evaluate interventions.

Work spread across many tools cannot be measured from any single product's analytics. This method normalises logs from all tools into events, tags artefacts as task-identifying (such as a code change) or informational, and groups events into sessions with explicit rules: same day, gaps under 10 minutes, same task artefact. Sessions are mapped to a small set of time metrics; at Google these were coding, reviewing, shepherding, investigation, development, email and meetings. Crucially, the metrics are validated: 25 engineers kept one-day diaries, and agreement with the logs was computed per metric with prevalence-adjusted bias-adjusted kappa (PABAK), benchmarked against well-understood measures such as meetings. Only validated metrics were used in mixed models with person random effects, for example showing readability certification associated with 4.5% shorter review time.

Attributes

Loss type
Latent
Data
Mixed
Effort
Months

Steps

  1. Inventory log sources and prioritise those covering most of the work.
  2. Normalise logs into events and tag artefacts as task-identifying or informational.
  3. Group events into sessions with explicit rules, for example same day, gap under 10 minutes and same task artefact.
  4. Map sessions to a small set of time metrics tied to recognisable activities.
  5. Collect diaries from a sample of participants (for example 25) for the same days and compute agreement per metric with PABAK, comparing against well-understood metrics such as meetings.
  6. Enrich sources where agreement is weak (for example add window-focus events) and set up automated monitoring of data quality.
  7. Apply privacy rules: no individual reporting without consent, limited retention.
  8. Use validated metrics in models such as mixed models with person random effects to evaluate interventions.

Sources

  • Quant UX Con 2022 Proceedings (Chapman et al., 2022)

Also known as

  • InSession
  • Log-derived time metrics validated by diaries

Categories

  • Analytics & Behaviour
  • Product Analytics & Strategy