一句话：提出 RankEF，用多任务学习将代码分类与执行反馈生成结合，无需执行代码即可对 LLM 生成的代码候选排序。
大语言模型虽能自动生成代码，但在复杂任务上仍难以保证语法与语义正确。常见做法是用 LLM 生成多个候选解再从中挑选，而「代码排序」问题依然困难：基于执行的方法受制于高质量单元测试稀缺与安全风险，非执行方法（如仅用分类标签训练的 CodeRanker）又难以捕捉细微错误、缺乏错误解释。本文提出 RankEF，其核心洞见是有效的排序器必须真正理解错误代码的成因，仅靠分类标签不足。方法上采用多任务学习，将代码分类与执行反馈生成联合训练，使模型在排序阶段无需真正执行代码即可区分正确与错误解。在 APPS、MBPP、HumanEval 三个代码生成基准上，RankEF 显著优于当时最优的 CodeRanker，在 APPS 测试集上 Pass@1、Pass@2、Pass@5 分别取得 +30.97%、+31.43%、+19.51% 的相对提升。
Large Language Models (LLMs), such as GPT-4, StarCoder, and Code Llama, are transforming the way developers approach programming by automatically generating code based on given contexts, such as natural language descriptions or incomplete surrounding code. Despite advancements, generating syntactically and semantically correct code remains challenging, especially for complex programming tasks. Existing approaches typically generate multiple candidate solutions using LLMs to increase the likelihood of producing correct code. However, selecting the correct code from these candidates --- a process known as code ranking --- remains a major challenge. Current research on code ranking can be categorized into execution-based and non-execution-based methods. Execution-based methods, although effective, encounter notable limitations, such as scarcity of quality unit tests and security risks. Non-execution-based methods like CodeRanker, which rely solely on classification labels to train a code ranker, struggle to capture subtle errors and provide detailed error insights. Recognizing the strengths and limitations of both approaches, we propose a new method that integrates the advantages of execution-based and non-execution-based techniques. The key insight of our work is that an effective code ranker is expected to truly comprehend the underlying causes of erroneous code, as relying solely on classification labels is insufficient. Inspired by this, this paper puts forward RankEF, an innovative approach for code ranking that leverages execution feedback. RankEF employs multi-task learning to integrate code classification with execution feedback generation. This approach enables the model to understand the reasons behind incorrect code, distinguishing between correct and incorrect solutions without the need to execute the code during the ranking phase. Experiments on three code generation benchmarks---APPS, MBPP, and HumanEval---demonstrate that RankEF significantly outperforms the state-of-the-art CodeRanker, achieving relative improvements of +30.97%, +31.43%, and +19.51% in Pass@1, Pass@2, and Pass@5 on APPS test, respectively.
DOI 原文 ·
@inproceedings{paperbot8323,
title = {Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates},
author = {Zhihong Sun and Yao Wan and Jia Li and Hongyu Zhang and Zhi Jin and Ge Li and Chen Lyu},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695000}
}