一句话：提出无监督日志解析方法 LibreLog，用开源 LLM 结合分组、RAG、自反思与模板记忆，兼顾隐私、成本与准确率。
日志解析将非结构化日志转为结构化格式，是后续日志分析的关键步骤。传统语法解析器高效，但处理偏离预定义规则的日志时准确率下降；现有基于大语言模型（LLM）的解析器虽准确，却依赖耗时人工标注、成本高，且商业模型有隐私风险。LibreLog 是无监督方法，基于开源 LLM（Llama3-8B），用固定深度分组树聚合静态文本相似而动态变量不同的日志，并通过相似度检索增强生成、自反思优化模板、日志模板记忆三项机制提升效果。在 LogHub-2.0 上，其准确率比先进 LLM 解析器高 25%，速度快 2.7 倍，同时缓解隐私与成本问题。
Log parsing is a critical step that transforms unstructured log data into structured formats, facilitating subsequent log-based analysis. Traditional syntax-based log parsers are efficient and effective, but they often experience decreased accuracy when processing logs that deviate from the predefined rules. Recently, large language models (LLM) based log parsers have shown superior parsing accuracy. However, existing LLM-based parsers face three main challenges: 1) time-consuming and labor-intensive manual labeling for fine-tuning or in-context learning, 2) increased parsing costs due to the vast volume of log data and limited context size of LLMs, and 3) privacy risks from using commercial models like ChatGPT with sensitive log information. To overcome these limitations, this paper introduces LibreLog, an unsupervised log parsing approach that leverages open-source LLMs (i.e., Llama3-8B) to enhance privacy and reduce operational costs while achieving state-of-the-art parsing accuracy. LibreLog first groups logs with similar static text but varying dynamic variables using a fixed-depth grouping tree. It then parses logs within these groups using three components: i) similarity scoring-based retrieval augmented generation: selects diverse logs within each group based on Jaccard similarity, helping the LLM distinguish between static text and dynamic variables; ii) self-reflection: iteratively query LLMs to refine log templates to improve parsing accuracy; and iii) log template memory: stores parsed templates to reduce LLM queries for improved parsing efficiency. Our evaluation on LogHub-2.0 shows that LibreLog achieves 25% higher parsing accuracy and processes logs 2.7 times faster compared to state-of-the-art LLM-based parsers. In short, LibreLog addresses privacy and cost concerns of using commercial LLMs while achieving state-of-the-arts parsing efficiency and accuracy.
DOI 原文 ·
@inproceedings{paperbot8825,
title = {LibreLog: Accurate and Efficient Unsupervised Log Parsing Using Open-Source Large Language Models},
author = {Zeyang Ma and Dong Jae Kim and Tse-Hsun Peter Chen},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00103}
}