Description
Turning raw clickstream events into theoretically meaningful per-session and per-user features, such as depth, dwell, backtracking and error recovery, for modelling behaviour.
Behavioural logs record events, but models and personas need features. Session feature engineering first groups events into sessions (commonly with a 30-minute inactivity timeout), then derives variables such as session length, number of distinct features used, depth into funnels, revisits, backtracking (returns to earlier steps), search-then-browse ratios, error events and time between sessions. Rigour comes from grounding each feature in a construct identified in qualitative work (for example 'hesitation' operationalised as dwell longer than the 90th percentile before a commit action), documenting definitions in a feature dictionary, handling bots and internal traffic, and checking feature distributions and correlations before clustering or modelling. Poorly chosen features determine poorly grounded segments.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Siloed
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Define the analytical unit (session, user-week, user) and the session timeout (30 minutes is conventional; validate against the inter-event time distribution).
- List target constructs from qualitative research and specify an operational feature for each in a feature dictionary.
- Remove bots, internal users and test accounts, and document the filters.
- Compute features with pandas group-by operations, log-transforming skewed counts and durations.
- Check distributions, missingness and pairwise correlations, dropping or combining features with correlations above about 0.85.
- Standardise features and validate a sample of feature values against session replays or raw event sequences.
Sources
- lifelines (survival)
- scikit-learn (GMM, LDA, UMAP)
- pandas (log parsing)
- matplotlib
- plotly
- Cooley, R., Mobasher, B. & Srivastava, J. (1999) Data preparation for mining World Wide Web browsing patterns. Knowledge and Information Systems
- Zheng, A. & Casari, A. (2018) Feature Engineering for Machine Learning
- The Inquiry Series, C4 — Behavioural Log Analysis — From Clickstreams to Personas
Also known as
- Behavioural feature extraction
- Sessionisation
Categories
- Analytics & Behaviour
- Computational & Machine Learning