Description
Train a sequence model on normal event logs to predict the next event, and flag events, timings or sessions that fall outside its likely predictions.
DeepLog treats a log as a language: each entry is parsed into an event key (its template) and a parameter vector that includes the time since the previous event. A stacked LSTM trained on windows of normal sessions learns which event usually follows the last few; during detection, an event that is not among the model's top few predictions marks an execution-path anomaly. For events that are normal, a per-event model forecasts parameter values such as elapsed time, and errors beyond a confidence interval fitted on validation data mark performance anomalies. A session is abnormal if any entry is flagged. False alarms reported by people are fed back to update the model online, and a workflow model built from the logs helps locate which step went wrong. On system logs it outperformed PCA and invariant-mining baselines.
Attributes
- Discipline
- Computational & Machine Learning
- Loss type
- Unmigrated
- Stage
- Post-launch
- Data
- Quantitative
- Effort
- Weeks
Steps
- Parse raw log entries into event keys and parameter vectors that include the elapsed time between events.
- Group entries into sessions using a session or object identifier.
- Train a stacked LSTM on windows of the last h events from normal sessions to predict the next event (the original work used small h and trained on under 1% of the data).
- During streaming detection, flag any observed event that is not among the top g predicted events.
- For normal events, forecast the parameter vector with a per-event model and flag errors outside a confidence interval fitted on validation errors.
- Mark a session as abnormal if any entry is flagged, and route flags to analysts with the surrounding sequence.
- Feed analyst-confirmed false positives back as training examples to update the model, and tune h and g to balance precision and recall.
Sources
- DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning (Du et al., 2017)
Also known as
- DeepLog
- Log key anomaly detection
- Next-event prediction anomaly detection
- Sequence-model anomaly detection on event logs
- LSTM session anomaly detection
Categories
- Analytics & Behaviour
- Computational & Machine Learning