一句话：提出 LogRobust，用日志事件语义向量与注意力 Bi-LSTM 检测不稳定日志中的异常。
日志被广泛用于大型复杂软件系统的故障排查，但已有基于日志的异常检测方法通常假设日志数据随时间稳定、不同日志事件集合已知。实践中，日志语句演化与日志数据处理噪声会导致出现未见事件或未见序列，使该封闭世界假设失效。本文提出 LogRobust：先提取日志事件的语义信息并表示为语义向量，再利用基于注意力的 Bi-LSTM 模型检测异常，以捕获日志序列中的上下文信息并自动学习不同日志事件的重要性，从而识别和处理不稳定的事件与序列。作者在 Hadoop 系统日志和 Microsoft 实际在线服务系统日志上评估，结果表明该方法能较好应对日志不稳定问题，在真实、持续变化的日志数据上取得准确且鲁棒的结果。
Logs are widely used by large and complex software-intensive systems for troubleshooting. There have been a lot of studies on log-based anomaly detection. To detect the anomalies, the existing methods mainly construct a detection model using log event data extracted from historical logs. However, we find that the existing methods do not work well in practice. These methods have the close-world assumption, which assumes that the log data is stable over time and the set of distinct log events is known. However, our empirical study shows that in practice, log data often contains previously unseen log events or log sequences. The instability of log data comes from two sources: 1) the evolution of logging statements, and 2) the processing noise in log data. In this paper, we propose a new log-based anomaly detection approach, called LogRobust. LogRobust extracts semantic information of log events and represents them as semantic vectors. It then detects anomalies by utilizing an attention-based Bi-LSTM model, which has the ability to capture the contextual information in the log sequences and automatically learn the importance of different log events. In this way, LogRobust is able to identify and handle unstable log events and sequences. We have evaluated LogRobust using logs collected from the Hadoop system and an actual online service system of Microsoft. The experimental results show that the proposed approach can well address the problem of log instability and achieve accurate and robust results on real-world, ever-changing log data.
DOI 原文 ·
@inproceedings{paperbot5080,
title = {Robust log-based anomaly detection on unstable log data},
author = {Xu Zhang and Yong Xu and Qingwei Lin and Bo Qiao and Hongyu Zhang and Yingnong Dang and Chunyu Xie and Xinsheng Yang and Qian Cheng and Ze Li and Junjie Chen and Xiaoting He and Randolph Yao and Jian-Guang Lou and Murali Chintalapati and Furao Shen and Dongmei Zhang},
booktitle = {FSE},
year = {2019},
doi = {10.1145/3338906.3338931}
}