一句话：提出 CYCLE 框架，让代码语言模型依据执行反馈自我精化错误生成，显著增强自修复能力。
预训练代码语言模型在代码生成上表现优异，但现有评测通常只关注一次性预测准确率，忽视了模型的自我精化能力。作者发现，代码语言模型难以高效地自行修正错误生成，而开发者调试并非自己编写的错误预测也十分困难。本文提出 CYCLE 框架，利用可用反馈（如测试套件报告的执行结果）学习对错误生成进行自我精化。作者在 HumanEval、MBPP 和 APPS 三个主流代码生成基准上评估 CYCLE，并实现参数量为 350M、1B、2B 和 3B 的四种变体。实验表明，CYCLE 能维持甚至提升一次性代码生成质量，同时显著增强代码语言模型的自我精化能力，持续提升代码生成性能，最高提升达 63.5。
Pre-trained code language models have achieved promising performance in code generation and improved the programming efficiency of human developers. However, their self-refinement capability is typically overlooked by the existing evaluations of code LMs, which focus only on the accuracy of the one-time prediction. For the cases when code LMs fail to implement the correct program, developers actually find it hard to debug and fix the faulty prediction since it is not written by the developers themselves. Unfortunately, our study reveals that code LMs cannot efficiently self-refine their faulty generations as well. In this paper, we propose CYCLE framework, learning to self-refine the faulty generation according to the available feedback, such as the execution results reported by the test suites. We evaluate CYCLE on three popular code generation benchmarks, HumanEval, MBPP, and APPS. The results reveal that CYCLE successfully maintains, sometimes improves, the quality of one-time code generation, while significantly improving the self-refinement capability of code LMs. We implement four variants of CYCLE with varied numbers of parameters across 350M, 1B, 2B, and 3B, and the experiments show that CYCLE consistently boosts the code generation performance, by up to 63.5
@article{paperbot2538,
title = {CYCLE: Learning to Self-Refine the Code Generation},
author = {Yangruibo Ding and Marcus J. Min and Gail Kaiser and Baishakhi Ray},
journal = {Proceedings of the ACM on Programming Languages},
volume = {8},
number = {OOPSLA1},
year = {2024},
doi = {10.1145/3649825}
}