一句话：提出 UniLog，用 LLM 的上下文学习实现无需微调的端到端自动日志语句生成，并显著降低调优成本。
日志语句的自动插入需要决定记录位置、日志级别与消息内容。现有工具多只覆盖单一子任务，端到端方法又受限于大模型微调成本。本文提出 UniLog，基于大语言模型的上下文学习（ICL）范式：仅需包含五个演示示例的提示即可生成合适的日志语句，无需任何模型调参；在少量随机样本上预热后还能进一步提升日志能力。作者在来自 1,465 个 GitHub 仓库的 12,012 个代码片段上评估。UniLog 在日志位置选择、详细级别预测和日志消息生成上分别达到 76.9%、72.3% 和 27.1 BLEU-4，并仅需同一 LLM 微调所需参数调优时间的不到 4%。
Logging, which aims to determine the position of logging statements, the verbosity levels, and the log messages, is a crucial process for software reliability enhancement. In recent years, numerous automatic logging tools have been designed to assist developers in one of the logging tasks (e.g., providing suggestions on whether to log in try-catch blocks). These tools are useful in certain situations yet cannot provide a comprehensive logging solution in general. Moreover, although recent research has started to explore end-to-end logging, it is still largely constrained by the high cost of fine-tuning, hindering its practical usefulness in software development. To address these problems, this paper proposes UniLog, an automatic logging framework based on the in-context learning (ICL) paradigm of large language models (LLMs). Specifically, UniLog can generate an appropriate logging statement with only a prompt containing five demonstration examples without any model tuning. In addition, UniLog can further enhance its logging ability after warmup with only a few hundred random samples. We evaluated UniLog on a large dataset containing 12,012 code snippets extracted from 1,465 GitHub repositories. The results show that UniLog achieved the state-of-the-art performance in automatic logging: (1) 76.9% accuracy in selecting logging positions, (2) 72.3% accuracy in predicting verbosity levels, and (3) 27.1 BLEU-4 score in generating log messages. Meanwhile, UniLog requires less than 4% of the parameter tuning time needed by fine-tuning the same LLM.
DOI 原文 ·
@inproceedings{paperbot11462,
title = {UniLog: Automatic Logging via LLM and In-Context Learning},
author = {Junjielong Xu and Ziang Cui and Yuan Zhao and Xu Zhang and Shilin He and Pinjia He and Liqun Li and Yu Kang and Qingwei Lin and Yingnong Dang and Saravan Rajmohan and Dongmei Zhang},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3623326}
}