一句话：提出 EXPO 框架，用 Bridge 与 Hint 两种记忆机制增强预训练语言模型对长代码的理解能力。
长距离代码建模是代码智能领域的难题：现有预训练语言模型（如 UniXcoder）虽有成效，但受限于维持上下文连续性与记忆关键信息的能力，难以处理长代码输入。本文提出 EXPO 框架，通过两种记忆机制扩展预训练语言模型：Bridge Memory 借助标注机制连接长距离代码中离散的片段，帮助模型维持上下文连贯；Hint Memory 关注全局上下文中的关键代码元素（如包导入），通过引入 kNN 注意力层自适应地选择相关代码元素。二者结合，弥合了局部代码片段理解与全局代码连贯性之间的鸿沟，提升模型对长代码序列的整体理解。作者在 UniXcoder 等五种预训练语言模型、以及 API 推荐与漏洞检测两个代码智能任务上验证了 EXPO 的有效性，实验结果表明其显著提升了预训练语言模型的性能。
In the field of code intelligence, effectively modeling long-range code poses a significant challenge. Existing pre-trained language models (PLMs) such as UniXcoder have achieved remarkable success, but they still face difficulties with long code inputs. This is mainly due to their limited capacity to maintain contextual continuity and memorize the key information over long-range code. To alleviate the difficulties, we propose EXPO, a framework for EXtending Pre-trained language models for lOng-range code. EXPO incorporates two innovative memory mechanisms we propose in this paper: Bridge Memory and Hint Memory. Bridge Memory uses a tagging mechanism to connect disparate snippets of long-range code, helping the model maintain contextual coherence. Hint Memory focuses on crucial code elements throughout the global context, such as package imports, by integrating a 𝑘NN attention layer to adaptively select the relevant code elements. This dual-memory approach bridges the gap between understanding local code snippets and maintaining global code coherence, thereby enhancing the model’s overall comprehension of long code sequences. We validate the effectiveness of EXPO on five popular pre-trained language models such as UniXcoder and two code intelligence tasks including API recommendation and vulnerability detection. Experimental results demonstrate that EXPO significantly improves the pre-training language models.
DOI 原文 ·
@inproceedings{paperbot8592,
title = {Bridge and Hint: Extending Pre-trained Language Models for Long-Range Code},
author = {Yujia Chen and Cuiyun Gao and Zezhou Yang and Hongyu Zhang and Qing Liao},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3652127}
}