一句话：提出三条评估日志模板识别准确性的指南，并据此重新评估14种技术，得出比以往更悲观的结论。
日志消息模板识别旨在将含自由格式日志消息的原始日志转换为结构化日志，以支持异常检测、模型推断等自动化日志分析。现有文献提出众多技术，但仅有两项近期研究基于真实日志基准进行了全面评估与比较。本文指出这两项研究存在三个问题：未对所用不同准确性指标进行比较、部分真值（oracle）模板有误、且准确性评估结果未提供关于错误识别模板的任何信息。为此，作者提出三条评估指南：使用恰当的准确性指标、进行真值模板校正、以及对错误模板进行分析。随后，他们依照这些指南，在既有基准日志上对14种现有模板识别技术开展了全面评估。结果表明，所得洞见与现有研究显著不同，尤其对现有技术的前景远不如以往乐观。
Log message template identification aims to convert raw logs containing free-formed log messages into structured logs to be processed by automated log-based analysis, such as anomaly detection and model inference. While many techniques have been proposed in the literature, only two recent studies provide a comprehensive evaluation and comparison of the techniques using an established benchmark composed of real-world logs. Nevertheless, we argue that both studies have the following issues: (1) they used different accuracy metrics without comparison between them, (2) some ground-truth (oracle) templates are incorrect, and (3) the accuracy evaluation results do not provide any information regarding incorrectly identified templates. In this paper, we address the above issues by providing three guidelines for assessing the accuracy of log template identification techniques: (1) use appropriate accuracy metrics, (2) perform oracle template correction, and (3) perform analysis of incorrect templates. We then assess the application of such guidelines through a comprehensive evaluation of 14 existing template identification techniques on the established benchmark logs. Results show very different insights than existing studies and in particular a much less optimistic outlook on existing techniques.
@inproceedings{Khan0BB22,
title = {Guidelines for assessing the accuracy of log message template identification techniques},
author = {Zanis Ali Khan and Donghwan Shin and Domenico Bianculli and Lionel Briand},
booktitle = {ICSE},
year = {2022},
doi = {10.1145/3510003.3510101}
}