一句话：系统实证微调范式下代码大模型的程序修复能力，覆盖多语言、多场景与多 hunk 缺陷，并显著超越现有 APR 工具。
论文关注自动程序修复（APR）中大代码语言模型（LLMC）的应用。已有工作多采用零样本/少样本范式，直接让模型依据上下文生成补丁，而微调范式的修复能力尚未被充分探索，模型能否修复多 hunk 等更复杂缺陷也不清楚。为此，作者选取 5 个具有代表性预训练架构的 LLMC（CodeBERT、GraphCodeBERT、PLBART、CodeT5、UniXcoder），覆盖 3 类修复场景（缺陷、漏洞、错误）与 3 种语言（Java、C/C++、JavaScript），并同时考虑单 hunk 与多 hunk 缺陷/漏洞。他们在广泛使用的数据集上对模型进行微调，与现有最先进 APR 工具对比，并考察代码抽象、代码表示与评价指标等设计选择的影响。实验表明，微调范式下的 LLMC 能显著优于此前最先进的 APR 工具；作者进一步给出选择合适策略以提升性能的洞见，并揭示其局限、提出未来研究方向。
The advent of large language models (LLMs) has opened up new opportunities for automated program repair (APR). In particular, some recent studies have explored how to leverage large language models of code (LLMCs) for program repair tasks and show promising results. However, most of them adopt the zero/few-shot learning paradigm for APR, which directly use LLMCs to generate the possibly correct code given its surrounding context. Though effective, the repair capabilities of LLMCs based on the fine-tuning paradigm have yet to be extensively explored. Also, it remains unknown whether LLMCs have the potential to repair more complicated bugs (e.g., multi-hunk bugs). To fill the gap, in this work, we conduct a comprehensive study on the program repair capability of LLMCs in the fine-tuning paradigm. We select 5 popular LLMCs with representative pre-training architectures, including CodeBERT, GraphCode-BERT, PLBART, CodeT5, and UniX coder. We consider 3 typical program repair scenarios (i.e., bugs, vulnerabilities, and errors) involving 3 programming languages (i.e., Java, $\mathrm{C}/\mathrm{C}++$, and JavaScript). Notably, we take both single-hunk and multi-hunk bugs/vulnerabilities into account. We then fine-tune them on widely-used datasets and compare them with existing state-of-the-art APR tools. We also investigate the impact of different design choices, which include code abstractions, code representations, and model evaluation metrics. Our experimental results show that LLMCs in the fine-tuning paradigm can significantly outperform previous state-of-the-art APR tools. Through in-depth analysis, we provide insights into choosing appropriate strategies to guide LLMCs for better performance. Lastly, we reveal several limitations of LLMCs for APR and make suggestions for future research on LLMC-based APR.
@inproceedings{HuangMZLWLZ23,
title = {An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair},
author = {Kai Huang and Xiangxin Meng and Jian Zhang and Yang Liu and Wenjie Wang and Shuhao Li and Yuqing Zhang},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00181}
}