一句话：提出 FT2Ra，通过模拟微调中的 delta logits、学习率与多轮检索，实现无需微调的检索增强代码补全。
代码预训练模型虽提升了代码补全等任务，但其规模使下游微调成本高昂；检索增强方法无需微调即可增强预测，但现有设计多依赖启发式，缺乏对存储/检索内容及插值方式的指导。本文先对微调过程进行理论分析，指出 delta logits 是提升模型预测的关键因素。基于此提出 FT2Ra：一种检索式方法，通过引入类似微调的学习率与多轮检索范式来模仿真实微调。作者在 token 级与行级代码补全上进行了全面评估。结果表明，FT2Ra 显著优于现有最佳方法：在较容易的 token 级补全中，于 UniXcoder 上准确率较最佳基线提升 4.29%；在更具挑战的行级补全中，Exact Match 提升超过两倍。值得注意的是，即使不进行真实微调，FT2Ra 也能与经过微调的模型竞争。
The rise of code pre-trained models has significantly enhanced various coding tasks, such as code completion, and tools like GitHub Copilot. However, the substantial size of these models, especially large models, poses a significant challenge when it comes to fine-tuning them for specific downstream tasks. As an alternative approach, retrieval-based methods have emerged as a promising solution, augmenting model predictions without the need for fine-tuning. Despite their potential, a significant challenge is that the designs of these methods often rely on heuristics, leaving critical questions about what information should be stored or retrieved and how to interpolate such information for augmenting predictions. To tackle this challenge, we first perform a theoretical analysis of the fine-tuning process, highlighting the importance of delta logits as a catalyst for improving model predictions. Building on this insight, we develop a novel retrieval-based method, FT2Ra, which aims to mimic genuine fine-tuning. While FT2Ra adopts a retrieval-based mechanism, it uniquely adopts a paradigm with a learning rate and multi-epoch retrievals, which is similar to fine-tuning. We conducted a comprehensive evaluation of FT2Ra in both token-level and line-level code completions. Our findings demonstrate the remarkable effectiveness of FT2Ra when compared to state-of-the-art methods and its potential to genuine fine-tuning. In token-level completion, which represents a relatively easier task, FT2Ra achieves a 4.29% improvement in accuracy compared to the best baseline method on UniXcoder. In the more challenging line-level completion task, we observe a substantial more than twice increase in Exact Match (EM) performance, indicating the significant advantages of our theoretical analysis. Notably, even when operating without actual fine-tuning, FT2Ra exhibits competitive performance compared to the models with real fine-tuning.
@inproceedings{paperbot8586,
title = {FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion},
author = {Qi Guo and Xiaohong Li and Xiaofei Xie and Shangqing Liu and Ze Tang and Ruitao Feng and Junjie Wang and Jidong Ge and Lei Bu},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3652130}
}