一句话：提出RRG框架，在检索器与生成器之间加入代码重构器，消除冗余并弥合偏好差距，提升代码生成效果与效率。
检索增强代码生成借助检索器为大语言模型提供相关代码与文档，但存在两个主要限制：一是无关冗余信息会被盲目引入，浪费资源并误导生成器；二是检索器与生成器的优化目标不同，导致两者偏好的“金标准”代码不一致，形成偏好差距，且不同生成器因预训练知识差异而有不同偏好。为此，本文提出RRG（Retrieve, Refactor, Generate）框架，在检索器与生成器之间引入代码重构器，将原始检索代码转换为更简洁、高效且模型友好的版本，去除冗余与噪声并缩短输入长度，从而以更低推理成本获得更高质量的上下文。多数据集实验证实了偏好差距的存在，并显示RRG能有效弥合该差距，最高提升EM 28%、BLEU 13%、CodeBLEU 6.8%。
Retrieval-augmented code generation utilizes Large Language Models as the generator and significantly expands their code generation capabilities by providing relevant code, documentation, and more via the retriever. The current approach suffers from two primary limitations: 1) information redundancy. The indiscriminate inclusion of redundant information can result in resource wastage and may misguide generators, affecting their effectiveness and efficiency. 2) preference gap. Due to different optimization objectives, the retriever strives to procure code with higher ground truth similarity, yet this effort does not substantially benefit the generator. The retriever and the generator may prefer different golden code, and this gap in preference results in a suboptimal design. Additionally, differences in parameterization knowledge acquired during pre-training result in varying preferences among different generators.To address these limitations, in this paper, we propose RRG (Retrieve, Refactor, Generate), a novel framework for effective and efficient code generation. This framework introduces a code refactorer module between the retriever and the generator to bridge them. The refactoring process transforms the raw retrieved code into a more concise, efficient, and model-friendly version. It eliminates redundant information and noise, reducing the input length. Consequently, the generator receives higher-quality context, enabling it to produce more accurate results with lower inference costs. We conducted comprehensive experiments on multiple datasets. In the experiments, we confirmed the existence of a preference gap between the retriever and the generator, and RRG effectively bridges this gap. Specifically, RRG achieved significant performance improvements, with increases of up to 28% on EM, 13% on BLEU, and 6.8% on CodeBLEU.
@inproceedings{abs-2409-15895,
title = {Preference-Guided Refactored Tuning for Retrieval Augmented Code Generation},
author = {Xinyu Gao and Yun Xiong and Deze Wang and Zhenhan Guan and Zejian Shi and Haofen Wang and Shanshan Li},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3694987}
}