一句话：提出 MMLoRA，用通用专家与语言专用专家的混合低秩适配，缓解多语言代码摘要中的梯度冲突。
代码语言模型（CLM）常用于多语言代码智能任务，但全参数微调耗时且耗资源，参数高效微调（PEFT）虽更高效，却难以捕捉跨语言共享特征，导致性能下降。已有工作尝试用混合语言训练配合 PEFT 来保留共性特征，但语言特有特征的多样性会引发梯度冲突，使性能次优，低资源语言尤为明显。本文提出面向多语言代码摘要的专家混合低秩适配 MMLoRA：将一个大语言无关的通用专家与一组语言专用专家相结合，在保留跨语言共享特征的同时缓解梯度冲突；并设计专家损失函数，在维持语言专用专家多样性的同时平衡各专家的学习进度。实验表明，MMLoRA 在多语言代码摘要任务上取得了当前最优性能，且保持高效微调，在 Ruby 等低资源语言上的提升尤为显著。
As Code Language Models (CLMs) are increasingly used to automate multilingual code intelligence tasks, Full-Parameter Fine-Tuning (FPFT) of CLMs has become a widely adopted approach, which is both time-consuming and resource-intensive. Parameter-Efficient Fine-Tuning (PEFT) provides a more efficient alternative to FPFT. However, it struggles to capture common features shared across languages, leading to performance degradation. Recent studies have explored mixed-language training with PEFT to avoid the loss of common features. However, these methods can result in gradient conflicts due to the diverse language-specific features, causing suboptimal performance, particularly for low-resource languages. In this paper, we propose Mixture-of-Experts Multilingual Low-Rank Adaptation (MMLoRA) for multilingual code summarization. MMLoRA addresses gradient conflicts while preserving common features shared across languages by combining a universal expert with a set of specialized linguistic experts. Additionally, we introduce an expert loss function that maintains the diversity of specialized linguistic experts while balancing the learning progress. Experimental results indicate that MMLoRA achieves state-of-the-art performance in multilingual code summarization while maintaining efficient fine-tuning. The performance improvement is particularly significant in low-resource languages such as Ruby.
DOI 原文 ·
@inproceedings{paperbot9391,
title = {Mixture-of-Experts Low-Rank Adaptation for Multilingual Code Summarization},
author = {Tianchen Yu and Li Yuan and Hailing Huang and Jiexin Wang and Yi Cai},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00186}
}