一句话：提出数据驱动的 CoLA 方法，通过构建长上下文仓库级补全数据集与两阶段训练，让 LLM 学会利用跨文件上下文。
大语言模型在仓库级代码补全中已展现良好效果，但作者发现模型难以充分利用跨文件上下文中的信息（如相关 API 与相似代码）。作者推测其根因在于预训练（依赖近邻上下文）与仓库级补全（需频繁关注长距离跨文件上下文）之间的错位。为此提出 CoLA（Code Long-context Alignment），一种纯数据驱动的方法：构建大规模仓库级补全数据集 CoLA-132K，每个样本包含最长 128K token 的跨文件上下文，并要求生成上下文感知的代码（跨文件 API 调用与与跨文件上下文相似的代码片段）；再通过两阶段训练流水线，使模型习得在跨文件上下文中定位相关信息的能力。作者将 CoLA 应用于 aiXcoder-7B 等多个流行模型，并在 CoLA-132K 与公开基准 CrossCodeEval 上开展实验：aiXcoder-7B 的 exact match 最高提升 19.7%；所学能力可泛化到训练数据中未出现的语言；两个探测实验表明模型利用跨文件上下文中相关信息（相关 API 与相似代码）的能力得到增强。
Large Language Models (LLMs) have shown promising results in repository-level code completion, which completes code based on the in-file and cross-file context of a repository. The cross-file context typically contains different types of information (e.g., relevant APIs and similar code) and is lengthy. In this paper, we found that LLMs struggle to fully utilize the information in the cross-file context. We hypothesize that one of the root causes of the limitation is the misalignment between pre-training (i.e., relying on nearby context) and repo-level code completion (i.e., frequently attending to long-range cross-file context).To address the above misalignment, we propose Code Long-context Alignment - CoLA, a purely data-driven approach to explicitly teach LLMs to focus on the cross-file context. Specifically, CoLA constructs a large-scale repo-level code completion dataset - CoLA-132K, where each sample contains the long cross-file context (up to 128K tokens) and requires generating context-aware code (i.e., cross-file API invocations and code spans similar to cross-file context). Through a two-stage training pipeline upon CoLA-132K, LLMs learn the capability of finding relevant information in the cross-file context, thus aligning LLMs with repo-level code completion. We apply CoLA to multiple popular LLMs (e.g., aiXcoder-7B) and extensive experiments on CoLA-132K and a public benchmark - CrossCodeEval. Our experiments yield the following results. ❶ Effectiveness. CoLA substantially improves the performance of multiple LLMs in repo-level code completion. For example, it improves aiXcoder-7B by up to 19.7% in exact match. ❷ Generalizability. The capability learned by CoLA can generalize to new languages (i.e., languages not in training data). ❸ Enhanced Context Utilization Capability. We design two probing experiments, which show CoLA improves the capability of LLMs in utilizing the information (i.e., relevant APIs and similar code) in cross-file context. Our datasets and model weights are released in [1].
DOI 原文 ·
@inproceedings{paperbot9328,
title = {Aligning LLMs to Fully Utilize the Cross-file Context in Repository-level Code Completion},
author = {Jia Li and Hao Zhu and Huanyu Liu and Xianjie Shi and He Zong and Yihong Dong and Kechi Zhang and Siyuan Jiang and Zhi Jin and Ge Li},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00125}
}