一句话：定义混合日志解析新问题，并提出首个通过用户反馈自适应的解析器 Hue。
日志解析是从半结构化日志中抽取模板并生成结构化日志的第一步，也是自动日志分析中最关键的环节。现有解析器存在两方面设计局限：一是不原生支持同时包含单行日志与多行日志（如 Java Exception、Hadoop Counters）的混合日志；二是难以在解析中融入领域知识，从而无法识别歧义 token。本文提出并定义了混合日志解析这一新研究问题，它是传统日志解析任务的超集，并给出首个面向该问题的用户自适应方案 Hue。Hue 借助 key casting table 将每条日志消息转换为特殊通配符序列，再通过行聚合与模式抽取确定日志类型；同时利用新颖的 merge-reject 策略有效吸收用户反馈，从而快速适应复杂且不断变化的日志模板。作者在三个混合日志数据集和 Loghub 中十六个广泛使用的单行日志数据集上评估 Hue，混合日志上的平均分组准确率可达 0.845，大幅超过现有解析器的最佳结果（平均 0.563），并在单行日志数据集上也取得 SOTA 表现。
Log parsing, which extracts log templates from semi-structured logs and produces structured logs, is the first and the most critical step in automated log analysis. While existing log parsers have achieved decent results, they suffer from two major limitations by design. First, they do not natively support hybrid logs that consist of both single-line logs and multi-line logs (Java Exception and Hadoop Counters). Second, they fall short in integrating domain knowledge in parsing, making it hard to identify ambiguous tokens in logs. This paper defines a new research problem, hybrid log parsing, as a superset of traditional log parsing tasks, and proposes Hue, the first attempt for hybrid log parsing via a user-adaptive manner. Specifically, Hue converts each log message to a sequence of special wildcards using a key casting table and determines the log types via line aggregating and pattern extracting. In addition, Hue can effectively utilize user feedback via a novel merge-reject strategy, making it possible to quickly adapt to complex and changing log templates. We evaluated Hue on three hybrid log datasets and sixteen widely-used single-line log datasets (Loghub). The results show that Hue achieves an average grouping accuracy of 0.845 on hybrid logs, which largely outperforms the best results (0.563 on average) obtained by existing parsers. Hue also exhibits SOTA performance on single-line log datasets.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2308-07085,
title = {Hue: A User-Adaptive Parser for Hybrid Logs},
author = {Junjielong Xu and Qiuai Fu and Zhouruixing Zhu and Yutong Cheng and Zhijing Li and Yuchi Ma and Pinjia He},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616260}
}