一句话：在三个代码智能任务上实证表明提示调优一致优于微调，低资源场景下提升尤为显著。
预训练模型在代码智能任务中已被证明有效，其典型流程是先在大规模无标注语料上预训练，再在下游任务上微调。然而预训练与下游任务的输入形式不同，难以充分利用预训练知识；同时微调性能严重依赖下游数据量，而实际中数据稀缺的场景十分常见。自然语言处理领域提出的提示调优（prompt tuning）通过在调优阶段插入提示来提供任务特定知识，可缓解上述问题，尤其有利于数据较少的任务。本文在代码智能任务上实证评估提示调优的使用与效果：在流行的预训练模型 CodeBERT 与 CodeT5 上进行提示调优，覆盖缺陷预测、代码摘要与代码翻译三个任务。实验结果显示，提示调优在这三个任务上一致优于微调；且在低资源场景下潜力很大，例如在代码摘要任务上使微调的 BLEU 平均提升超过 26%。作者据此建议，可以用提示调优替代微调来适配代码智能任务，以取得更好性能，尤其是在缺乏任务特定数据时。
Pre-trained models have been shown effective in many code intelligence tasks. These models are pre-trained on large-scale unlabeled corpus and then fine-tuned in downstream tasks. However, as the inputs to pre-training and downstream tasks are in different forms, it is hard to fully explore the knowledge of pre-trained models. Besides, the performance of fine-tuning strongly relies on the amount of downstream data, while in practice, the scenarios with scarce data are common. Recent studies in the natural language processing (NLP) field show that prompt tuning, a new paradigm for tuning, alleviates the above issues and achieves promising results in various NLP tasks. In prompt tuning, the prompts inserted during tuning provide task-specific knowledge, which is especially beneficial for tasks with relatively scarce data. In this paper, we empirically evaluate the usage and effect of prompt tuning in code intelligence tasks. We conduct prompt tuning on popular pre-trained models CodeBERT and CodeT5 and experiment with three code intelligence tasks including defect prediction, code summarization, and code translation. Our experimental results show that prompt tuning consistently outperforms fine-tuning in all three tasks. In addition, prompt tuning shows great potential in low-resource scenarios, e.g., improving the BLEU scores of fine-tuning by more than 26% on average for code summarization. Our results suggest that instead of fine-tuning, we could adapt prompt tuning for code intelligence tasks to achieve better performance, especially when lacking task-specific data.
@inproceedings{paperbot7094,
title = {No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence},
author = {Chaozheng Wang and Yuanhang Yang and Cuiyun Gao and Yun Peng and Hongyu Zhang and Michael R. Lyu},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549113}
}