一句话：构建大规模标注日志数据集 Loghub-2.0，并重新评估 15 种日志解析器，揭示其实际性能与不足。
日志解析是将非结构化日志转为结构化数据以支持测试、调试与诊断的关键步骤。现有基准 Loghub 规模与代表性不足，难以全面评估日志解析器在生产环境中的表现。本文提出 Loghub-2.0，包含 14 个数据集，平均每个约 360 万行日志，更贴近真实系统。基于该数据集，作者在严格且实际的设定下重新评估 15 种先进日志解析器，并提出可缓解数据不平衡敏感性的新指标；同时首次考察解析器在罕见系统事件日志上的细粒度性能。结果表明，现有解析器在真实大规模与不平衡场景下仍面临显著挑战，尤其难以准确解析罕见事件日志，为生产部署与后续设计提供依据。
Log data have facilitated various tasks of software development and maintenance, such as testing, debugging and diagnosing. Due to the unstructured nature of logs, log parsing is typically required to transform log messages into structured data for automated log analysis. Given the abundance of log parsers that employ various techniques, evaluating these tools to comprehend their characteristics and performance becomes imperative. Loghub serves as a commonly used dataset for benchmarking log parsers, but it suffers from limited scale and representativeness, posing significant challenges for studies to comprehensively evaluate existing log parsers or develop new methods. This limitation is particularly pronounced when assessing these log parsers for production use. To address these limitations, we provide a new collection of annotated log datasets, denoted Loghub-2.0, which can better reflect the characteristics of log data in real-world software systems. Loghub-2.0 comprises 14 datasets with an average of 3.6 million log lines in each dataset. Based on Loghub-2.0, we conduct a thorough re-evaluation of 15 state-of-the-art log parsers in a more rigorous and practical setting. Particularly, we introduce a new evaluation metric to mitigate the sensitivity of existing metrics to imbalanced data distributions. We are also the first to investigate the granular performance of log parsers on logs that represent rare system events, offering in-depth details for software diagnosis. Accurately parsing such logs is essential, yet it remains a challenge. We believe this work could shed light on the evaluation and design of log parsers in practical settings, thereby facilitating their deployment in production systems.
DOI 原文 ·
@inproceedings{paperbot8496,
title = {A Large-Scale Evaluation for Log Parsing Techniques: How Far Are We?},
author = {Zhihan Jiang and Jinyang Liu and Junjie Huang and Yichen Li and Yintong Huo and Jiazhen Gu and Zhuangbin Chen and Jieming Zhu and Michael R. Lyu},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3652123}
}