Synthetic session generation at cluster boundaries

Description

Uses a generative model to create synthetic user sessions in the ambiguous regions between behavioural clusters, stress-testing systems on the users they are most likely to misclassify.

AI systems such as recommenders, risk models and adaptive interfaces tend to fail on users who sit between typical patterns. This method trains a generative model (often a VAE) on real sessions, identifies cluster boundaries in latent space (where membership probabilities are near equal), and samples or interpolates synthetic sessions there. These sessions are fed to the target system to observe outputs, measure inconsistency, and find failure modes before real users encounter them. It borrows from boundary-value analysis in software testing. Rigour requires validating that synthetic sessions are realistic, labelling them as synthetic throughout, and confirming discovered failures with real users or real logs. Synthetic sessions test systems; they do not replace user research.

Attributes

Loss type
Latent
Stage
Evaluation
Effort
Weeks

Steps

  1. Train a generative model on real sessions and fit clusters in its latent space.
  2. Identify boundary regions where the top two cluster probabilities differ by less than 0.2.
  3. Sample or interpolate thousands of latent points in those regions and decode them into synthetic sessions (e.g. 10,000).
  4. Validate realism by comparing synthetic sessions with real boundary users on key distributions and expert review of 30-50 samples.
  5. Run synthetic sessions through the target system and measure outputs: recommendation stability, risk score variance, inappropriate suggestions.
  6. Confirm high-severity failures with real user data or targeted research before acting.

Sources

  • Deep Generative Modeling (Jakub Tomczak, Springer)
  • Deep Learning (Ian Goodfellow, Yoshua Bengio & Aaron Courville)
  • Methods Atlas (Vivek Gopinath)
  • Kingma & Welling (2013) Auto-encoding variational Bayes
  • Deep Generative Modelling (Springer)
  • Deep Learning (Goodfellow)
  • Methods Atlas (Gopinath)
  • The Inquiry Series, Ch 53 — Generative Models & Synthetic User Testing

Also known as

  • Boundary-case synthetic testing
  • Edge-case synthetic users

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning