一句话：提出 LogCraft，首个基于 AutoML 的端到端无监督日志异常检测框架，自动化特征工程与模型选择。
现代软件系统日益复杂，日志分析对可靠运行至关重要，但现有日志异常检测方法依赖专家人工调参，跨系统迁移性差：在新数据集上部署有效算法常需深入理解数据并反复尝试。为此，论文提出 LogCraft，一个基于 AutoML 的端到端无监督日志异常检测框架，自动完成特征工程、模型选择与异常检测，降低专业知识门槛与部署成本。论文还从完整日志数据集提取固定维向量作为潜在表示，并用元特征提取器度量日志数据集相似性。在五个公开数据集上，LogCraft 平均 F1 达 0.899，优于现有无监督算法的次优平均 F1 0.847，验证了其有效性与迁移潜力。
As modern software systems evolve towards greater complexity, ensuring their reliable operation has become a critical challenge. Log data analysis is vital in maintaining system stability, with anomaly detection being a key aspect. However, existing log anomaly detection methods heavily rely on manual effort from experts, lacking transferability across systems. This has led to the situation where to perform anomaly detection on a new dataset, the operators must have a high level of understanding of the dataset, make multiple attempts, and spend a lot of time to deploy an algorithm that performs well successfully. This paper proposes LogCraft, an end-to-end unsupervised log anomaly detection framework based on automated machine learning (AutoML). LogCraft automates feature engineering, model selection, and anomaly detection, reducing the need for specialized knowledge and lowering the threshold for algorithm deployment. Extensive evaluations on five public datasets demonstrate LogCraft's effectiveness, achieving an average F1 score of 0.899, which outperforms the second-best average F1 score of 0.847 obtained by existing unsupervised algorithms. According to our knowledge, LogCraft is the first attempt to extract fixed-dimensional vectors as latent representations from a complete log dataset. The proposed meta-feature extractor also exhibits promising potential for measuring log dataset similarity and guiding future log analytics research.
DOI 原文 ·
@inproceedings{paperbot8398,
title = {End-to-End AutoML for Unsupervised Log Anomaly Detection},
author = {Shenglin Zhang and Yuhe Ji and Jiaqi Luan and Xiaohui Nie and Zi`ang Chen and Minghua Ma and Yongqian Sun and Dan Pei},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695535}
}