LSTM Next-Event Anomaly Detection on Event Logs (DeepLog)

Description

Train a sequence model on normal event logs to predict the next event, and flag events, timings or sessions that fall outside its likely predictions.

DeepLog treats a log as a language: each entry is parsed into an event key (its template) and a parameter vector that includes the time since the previous event. A stacked LSTM trained on windows of normal sessions learns which event usually follows the last few; during detection, an event that is not among the model's top few predictions marks an execution-path anomaly. For events that are normal, a per-event model forecasts parameter values such as elapsed time, and errors beyond a confidence interval fitted on validation data mark performance anomalies. A session is abnormal if any entry is flagged. False alarms reported by people are fed back to update the model online, and a workflow model built from the logs helps locate which step went wrong. On system logs it outperformed PCA and invariant-mining baselines.

Attributes

Loss type
Unmigrated
Stage
Post-launch
Effort
Weeks

Steps

  1. Parse raw log entries into event keys and parameter vectors that include the elapsed time between events.
  2. Group entries into sessions using a session or object identifier.
  3. Train a stacked LSTM on windows of the last h events from normal sessions to predict the next event (the original work used small h and trained on under 1% of the data).
  4. During streaming detection, flag any observed event that is not among the top g predicted events.
  5. For normal events, forecast the parameter vector with a per-event model and flag errors outside a confidence interval fitted on validation errors.
  6. Mark a session as abnormal if any entry is flagged, and route flags to analysts with the surrounding sequence.
  7. Feed analyst-confirmed false positives back as training examples to update the model, and tune h and g to balance precision and recall.

Sources

  • DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning (Du et al., 2017)

Also known as

  • DeepLog
  • Log key anomaly detection
  • Next-event prediction anomaly detection
  • Sequence-model anomaly detection on event logs
  • LSTM session anomaly detection

Categories

  • Analytics & Behaviour
  • Computational & Machine Learning