一句话：提出 ICR 检索器，用树损失和 LLM 反馈选取示例，显著提升代码摘要、程序合成与缺陷修复的上下文学习效果。
在代码智能任务中，基于大语言模型的上下文学习依赖示例选择。现有 BM25 等检索方法难以理解查询的语义与结构，且对不同领域用户查询适应性不足。本文提出 Instructive Code Retriever (ICR)，通过树基损失函数学习代码语料的语义与结构信息，并利用 LLM 对示例的反馈指导检索器训练，以更好地建模查询与示例的相关性。在代码摘要、程序合成和缺陷修复任务上评估，ICR 显著优于 SOTA：两个代码摘要数据集 BLEU-4 分别提升 50.0% 和 90.0%，程序合成 CodeBLEU 提升 74.6%，两个缺陷修复数据集 BLEU-4 提升 3.6 和 3.2。
Recent studies proposed to leverage large language models (LLMs) with In-Context Learning (ICL) to handle code intelligence tasks without fine-tuning. ICL employs task instructions and a set of examples as demonstrations to guide the model in generating accurate answers without updating its parameters. While ICL has proven effective for code intelligence tasks, its performance heavily relies on the selected examples. Previous work has achieved some success in using BM25 to retrieve examples for code intelligence tasks. However, existing approaches lack the ability to understand the semantic and structural information of queries, resulting in less helpful demonstrations. Moreover, they do not adapt well to the complex and dynamic nature of user queries in diverse domains. In this paper, we introduce a novel approach named Instructive Code Retriever (ICR), which is designed to retrieve examples that enhance model inference across various code intelligence tasks and datasets. We enable ICR to learn the semantic and structural information of the corpus by a tree-based loss function. To better understand the correlation between queries and examples, we incorporate the feedback from LLMs to guide the training of the retriever. Experimental results demonstrate that our retriever significantly outperforms state-of-the-art approaches. We evaluate our model's effectiveness on various tasks, i.e., code summarization, program synthesis, and bug fixing. Compared to previous state-of-the-art algorithms, our method achieved improvements of 50.0% and 90.0% in terms of BLEU-4 for two code summarization datasets, 74.6% CodeBLEU on program synthesis dataset, and increases of 3.6 and 3.2 BLEU-4 on two bug fixing datasets.
DOI 原文 ·
@inproceedings{paperbot8374,
title = {Instructive Code Retriever: Learn from Large Language Model's Feedback for Code Intelligence Tasks},
author = {Jiawei Lu and Haoye Wang and Zhongxin Liu and Keyu Liang and Lingfeng Bao and Xiaohu Yang},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3694997}
}