一句话：提出基于生成式LLM与少样本调优的日志解析器LLMParser，在16个开源系统上平均准确率达96%。
日志解析是从非结构化日志中提取结构化信息的关键步骤，但传统解析器受日志格式多样性影响，准确率有限并影响下游分析。本文探索使用大型语言模型进行日志解析，提出LLMParser：基于生成式LLM与少样本调优，采用Flan-T5-small、Flan-T5-base、LLaMA-7B和ChatGLM-6B四个模型。在16个开源系统上的评估显示，LLMParser平均解析准确率达96%，显著优于当前最优解析器。文章还系统分析训练规模、模型规模和预训练来源的影响：较小LLM可能更有效，如Flan-T5-base达到与LLaMA-7B相当的结果且推理更快；使用其他系统日志预训练并不总能提升准确率，Flan-T5-base有提升，而LLaMA的组准确率下降约55%。该研究为LLM用于日志解析提供了实证依据，并指出其局限与未来方向。
Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.
@inproceedings{paperbot11355,
title = {LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing},
author = {Zeyang Ma and An Ran Chen and Dong Jae Kim and Tse-Hsun Chen and Shaowei Wang},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639150}
}