一句话：系统研究 LLM 代码摘要工作流，评估自动评测、提示技术、解码参数、语言类型与模型能力差异。
本文对 LLM 时代的代码摘要进行系统全面研究，覆盖评测、提示、模型设置、语言类型与模型比较等环节。作者首先考察自动评测方法，发现 GPT-4 评测与人工评价最一致；随后比较 zero-shot、few-shot、chain-of-thought、critique 和 expert 五种提示技术，发现高级提示未必优于简单 zero-shot。实验还分析 top_p 与 temperature 的影响，发现其作用因基础模型和编程语言而异，但两者影响相似。进一步测试不同编程语言上的摘要能力，发现 LLM 在逻辑编程语言上的表现弱于过程式或面向对象语言。最后意外发现 7B 参数的 CodeLlamaInstruct 在生成设计理由和代码属性断言方面可超过 GPT-4。
To support software developers in understanding and maintaining programs, various automatic (source) code summarization techniques have been proposed to generate a concise natural language summary (i.e., comment) for a given code snippet. Recently, the emergence of large language models (LLMs) has led to a great boost in the performance of coderelated tasks. In this paper, we undertake a systematic and comprehensive study on code summarization in the era of LLMs, which covers multiple aspects involved in the workflow of LLMbased code summarization. Specifically, we begin by examining prevalent automated evaluation methods for assessing the quality of summaries generated by LLMs and find that the results of the GPT-4 evaluation method are most closely aligned with human evaluation. Then, we explore the effectiveness of five prompting techniques (zero-shot, few-shot, chain-of-thought, critique, and expert) in adapting LLMs to code summarization tasks. Contrary to expectations, advanced prompting techniques may not outperform simple zero-shot prompting. Next, we investigate the impact of LLMs' model settings (including top_p and temperature parameters) on the quality of generated summaries. We find the impact of the two parameters on summary quality varies by the base LLM and programming language, but their impacts are similar. Moreover, we canvass LLMs' abilities to summarize code snippets in distinct types of programming languages. The results reveal that LLMs perform suboptimally when summarizing code written in logic programming languages compared to other language types (e.g., procedural and object-oriented programming languages). Finally, we unexpectedly find that CodeLlamaInstruct with 7B parameters can outperform advanced GPT-4 in generating summaries describing code design rationale and asserting code properties. We hope that our findings can provide a comprehensive understanding of code summarization in the era of LLMs.
DOI 原文 ·
@inproceedings{paperbot8695,
title = {Source Code Summarization in the Era of Large Language Models},
author = {Weisong Sun and Yun Miao and Yuekang Li and Hongyu Zhang and Chunrong Fang and Yi Liu and Gelei Deng and Yang Liu and Zhenyu Chen},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00034}
}