一句话：首次系统评估 Mamba 类预训练模型在代码智能任务上的效果与效率，显示其总体优于 Transformer 基线。
Transformer 及其注意力机制是预训练语言模型的基础，但二次复杂度带来扩展性挑战。Mamba、Mamba-2 等次二次架构虽受关注，其在代码智能任务中的有效性仍缺乏系统研究。本文首次对基于 Mamba 的预训练语言模型进行系统实证研究，覆盖代码补全、代码生成和代码克隆检测三类典型任务。作者分别在代码上预训练了两个基于 Mamba 和 Mamba-2 的模型，并与典型 Transformer 模型（如 CodeGPT）在全量微调（FT）和参数高效微调（PEFT）设置下比较。实验表明，基于 Mamba 的模型在所有代码任务上总体更优；从零预训练架构分析、低资源分析以及最大 7B 规模的扩展对比进一步显示其在效果与效率上的优势。结果还表明，除架构外，预训练语料和任务本身也会显著影响代码建模性能。
The Transformer architecture and its core attention mechanism form the foundation of Pre-trained Language Models (PLMs) and have driven their remarkable progress across a wide range of code intelligence tasks. However, the quadratic complexity inherent in the attention mechanism poses scalability challenges. Recently, sub-quadratic architectures such as Mamba and Mamba-2 have emerged as compelling alternatives to the Transformer. While they have shown promising results and attracted increasing academic interest, their effectiveness in code intelligence tasks has not yet been fully explored.To fill this gap, we present the first systematic empirical study of Mamba-based PLMs on three typical code tasks (i.e., code completion, code generation, and code clone detection), covering both the code comprehension and generation categories to delve into their effectiveness and efficiency. We first pre-train two Mamba-based PLMs on code based on Mamba and Mamba-2, respectively. Subsequently, we evaluate these four PLMs against typical Transformer-based PLMs (e.g., CodeGPT) with Full fine-Tuning (FT) and Parameter-Efficient Fine-Tuning (PEFT) settings, demonstrating the overall superiority of Mamba-based PLMs across all code tasks. Subsequent experiments involve the architecture analysis via pre-training from scratch to isolate the influence of the training corpora and low-resource analysis via deliberately limiting the fine-tuning data volume. All demonstrate the superiority of Mamba-based PLMs in both efficacy and efficiency. Finally, we also extend the sizes of PLMs to larger scales (7B at most) and make comparisons with more diverse PLMs/LLMs. Experimental results demonstrate that pre-training corpora and tasks also heavily affect the code modeling performance, apart from architectures. This work provides a comprehensive investigation into Mamba-based PLMs in the context of code intelligence, uncovering their strengths, limitations, and potential for future applications.
DOI 原文 ·
@inproceedings{paperbot9446,
title = {Can Mamba Be Better? An Experimental Evaluation of Mamba in Code Intelligence},
author = {Shuo Liu and Jacky Keung and Zhen Yang and Zhenyu Mao and Yicheng Sun},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00155}
}