一句话：系统研究自动程序修复技术能否修复大型语言模型（如 Codex）在编程竞赛中生成的错误代码。
大型语言模型（如 Codex）能生成代码，但成功率低，尤其在复杂任务上，原因之一是模型缺乏程序语义意识，产生错误甚至无法编译的程序。本文系统研究自动程序修复（APR）技术能否修复语言模型在 LeetCode 竞赛中产生的错误解，以提升其代码可靠性。研究利用统计故障定位提供缺陷位置，并比较 Codex 编辑模式与现有 Java 修复工具 TBar、Recoder。结果发现：自动生成代码与人工编写的解法共享常见编程错误，说明 APR 有潜力修复自动生成代码；在给定故障定位信息时，Codex 编辑模式与 TBar、Recoder 相当或更好。基于实验分析，作者建议增强 APR 工具以突破补丁空间限制，未来 APR 工具应从增加修复模式转向基于合成/语义的方法，并探索结合语言模型与 APR 来策展补丁成分。
Large language models such as Codex, have shown the capability to produce code for many programming tasks. However, the success rate of existing models is low, especially for complex programming tasks. One of the reasons is that language models lack awareness of program semantics, resulting in incorrect programs, or even programs which do not compile. In this paper, we systematically study whether automated program repair (APR) techniques can fix the incorrect solutions produced by language models in LeetCode contests. The goal is to study whether APR techniques can enhance reliability in the code produced by large language models. Our study revealed that: (1) automatically generated code shares common programming mistakes with human-crafted solutions, indicating APR techniques may have potential to fix auto-generated code; (2) given bug location information provided by a statistical fault localization approach, the newly released Codex edit mode, which supports editing code, is similar to or better than existing Java repair tools TBar and Recoder in fixing incorrect solutions. By analyzing the experimental results generated by these tools, we provide several suggestions: (1) enhancing APR tools to surpass limitations in patch space (e.g., introducing more flexible fault localization) is desirable; (2) as large language models can derive more fix patterns by training on more data, future APR tools could shift focus from adding more fix patterns to synthesis/semantics based approaches, (3) combination of language models with APR to curate patch ingredients, is worth studying.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{FanGMRT23,
title = {Automated Repair of Programs from Large Language Models},
author = {Zhiyu Fan and Xiang Gao and Martin Mirchev and Abhik Roychoudhury and Shin Hwei Tan},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00128}
}