一句话：提出 SPINE 可扩展日志解析器，通过并行调度与用户反馈提升大规模不平衡日志的解析效率与演化适应能力。
日志解析从日志中抽取模板与参数，是自动化日志分析的关键前提。现有解析器在公开数据集上精度较好，但在工业场景下面临两大问题：海量且极不平衡的日志难以扩展，且缺乏用户反馈，无法在日志持续演化时快速调优。为此，作者提出 SPINE：初始分组与渐进聚类结合的日志解析器，并设计面向大规模不平衡日志的日志数据调度算法以提升并行化效率，同时引入用户反馈使解析器快速适应演化日志。在 16 个公开日志数据集上，SPINE 平均解析准确率超过 0.90，解析效率最高，优于现有最先进解析器；在微软生产环境中，16 个执行器不到 8 分钟解析 3000 万条日志，接近实时；在日志演化下，借助适量用户反馈可持续保持良好准确率。
Log parsing, which extracts log templates and parameters, is a critical prerequisite step for automated log analysis techniques. Though existing log parsers have achieved promising accuracy on public log datasets, they still face many challenges when applied in the industry. Through studying the characteristics of real-world log data and analyzing the limitations of existing log parsers, we identify two problems. Firstly, it is non-trivial to scale a log parser to a vast number of logs, especially in real-world scenarios where the log data is extremely imbalanced. Secondly, existing log parsers overlook the importance of user feedback, which is imperative for parser fine-tuning under the continuous evolution of log data. To overcome the challenges, we propose SPINE, which is a highly scalable log parser with user feedback guidance. Based on our log parser equipped with initial grouping and progressive clustering,we propose a novel log data scheduling algorithm to improve the efficiency of parallelization under the large-scale imbalanced log data. Besides, we introduce user feedback to make the parser fast adapt to the evolving logs. We evaluated SPINE on 16 public log datasets. SPINE achieves more than 0.90 parsing accuracy on average with the highest parsing efficiency, which outperforms the state-of-the-art log parsers. We also evaluated SPINE in the production environment of Microsoft, in which SPINE can parse 30million logs in less than 8 minutes under 16 executors, achieving near real-time performance. In addition, our evaluations show that SPINE can consistently achieve good accuracy under log evolution with a moderate number of user feedback.
DOI 原文 ·
@inproceedings{paperbot7034,
title = {SPINE: a scalable log parser with feedback guidance},
author = {Xuheng Wang and Xu Zhang and Liqun Li and Shilin He and Hongyu Zhang and Yudong Liu and Lingling Zheng and Yu Kang and Qingwei Lin and Yingnong Dang and Saravanakumar Rajmohan and Dongmei Zhang},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549176}
}