Description
Converting raw event or server logs into a clean, typed, ordered event table with consistent user identifiers, timestamps and event names.
Product logs arrive as JSON event streams, server logs or analytics exports with inconsistent schemas, time zones, duplicate events and renamed events across releases. Log parsing builds a reliable event table: parse each record, normalise timestamps to UTC, resolve user identity across devices where consented, map event names through a versioned taxonomy, deduplicate and order events. Rigour lies in validating the output against known totals (for example daily active users from the analytics platform), quantifying dropped or malformed records, and documenting schema changes, because an unnoticed instrumentation change can masquerade as a behaviour change.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Siloed
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Days
Steps
- Obtain the event schema and instrumentation change history from engineering before parsing.
- Parse records into a typed table (user_id, session_id, timestamp, event, properties) using pandas or a streaming parser for large files.
- Normalise all timestamps to UTC and order events within user by timestamp and sequence number.
- Map raw event names to a versioned taxonomy so renamed events are unified across releases.
- Remove exact duplicates and report the percentage of malformed or dropped records (investigate if above about 1%).
- Reconcile daily totals with the source analytics platform within an agreed tolerance before analysis.
Sources
- lifelines (survival)
- scikit-learn (GMM, LDA, UMAP)
- pandas (log parsing)
- matplotlib
- plotly
- Srivastava, J. et al. (2000) Web usage mining: discovery and applications of usage patterns from Web data. SIGKDD Explorations
- McKinney, W. (2022) Python for Data Analysis, 3rd ed.
- The Inquiry Series, C4 — Behavioural Log Analysis — From Clickstreams to Personas
Also known as
- Event log parsing
- Clickstream preprocessing
Categories
- Analytics & Behaviour
- Computational & Machine Learning