一句话：冻结预训练参数、仅微调 0.6% 的 adapter，在代码搜索与摘要上取得一致提升并缓解多语言微调的灾难性遗忘。
论文关注代码智能模型的多语言适配问题。已有工作声称多语言联合微调有益，但作者发现在 UniXcoder 和 CodeT5 等新模型上，多语言微调反而造成性能下降，原因可能是灾难性遗忘。为此，作者固定全部预训练参数，插入参数高效的 adapter 结构并仅对 adapter 进行微调，可训练参数仅为逐语言全模型微调的 0.6%。在代码搜索与代码摘要任务上，该方法取得一致的性能提升并达到当时最优结果；实验还验证了其在跨语言与低资源场景下的有效性——每个编程语言仅用 200 个样本的多语言微调即可接近使用全量数据的代码摘要效果。三个探测任务进一步表明，adapter 微调显著优于全模型微调，并能有效克服灾难性遗忘。
As pre-trained models automate many code intel-ligence tasks, a widely used paradigm is to fine-tune a model on the task dataset for each programming language. A recent study reported that multilingual fine-tuning benefits a range of tasks and models. However, we find that multilingual fine-tuning leads to performance degradation on recent models UniXcoder and CodeT5. To alleviate the potentially catastrophic forgetting issue in multilingual models, we fix all pre-trained model parameters, insert the parameter-efficient structure adapter, and fine-tune it. Updating only 0.6% of the overall parameters compared to full-model fine-tuning for each programming language, adapter tuning yields consistent improvements on code search and sum-marization tasks, achieving state-of-the-art results. In addition, we experimentally show its effectiveness in cross-lingual and low-resource scenarios. Multilingual fine-tuning with 200 samples per programming language approaches the results fine-tuned with the entire dataset on code summarization. Our experiments on three probing tasks show that adapter tuning significantly outperforms full-model fine-tuning and effectively overcomes catastrophic forgetting.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2303-15822,
title = {One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization},
author = {Deze Wang and Boxing Chen and Shanshan Li and Wei Luo and Shaoliang Peng and Wei Dong and Xiangke Liao},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00013}
}