一句话：提出首个面向 Rust 的仓库级上下文增量代码翻译基准 RustRepoTrans，含 375 个任务与细粒度评测框架。
现有代码翻译评测多集中于无依赖的函数级任务，或直接进行完整仓库翻译而超出模型能力，也未覆盖将新增/修改模块增量翻译到 Rust 等低资源目标语言的场景。本文提出 RustRepoTrans，首个面向增量翻译的仓库级上下文代码翻译基准，包含 375 个从 C、Java、Python 翻译到 Rust 的任务，并设计更细粒度的评测指标与增强评测框架。作者评测了 7 个代表性 LLM 并进行错误分析：DeepSeek-R1 表现最佳，Pass@1 为 51.5%，在基本功能、噪声鲁棒性和语法差异识别上均较好；但相比无仓库上下文基准，其 Pass@1 从 73.7% 降至 51.5%，下降 22.2 个百分点，表明仓库级上下文仍是显著挑战。
Recent advancements in large language models (LLMs) have demonstrated impressive capabilities in code translation, typically evaluated using benchmarks like CodeTransOcean and RepoTransBench. However, dependency-free benchmarks fail to capture real-world complexities by focusing primarily on simple function-level translations and overlooking repository-level context (e.g., dependencies). Full-repository translation benchmarks significantly exceed the current capabilities of existing models, resulting in performance bottlenecks that fail to provide actionable insights for guiding model development. Furthermore, existing benchmarks do not account for the scenario of incrementally translating new or modified modules from the source to the target language, which demands careful handling of repository-level contexts such as dependencies, cross-module references, and architectural divergence. Moreover, LLMs’ effectiveness in translating to newer, low-resource languages like Rust remains largely underexplored.To address these gaps, we introduce RustRepoTrans, the first repository-level context code translation benchmark targeting incremental translation, comprising 375 tasks translating into Rust from C, Java, and Python. Using this benchmark, we evaluate seven representative LLMs, analyzing their errors to assess limitations in complex translation scenarios. Among them, DeepSeek-R1 performs best with 51.5% Pass@1, excelling in both basic functionality and additional translation abilities, such as noise robustness and syntactical difference identification. However, even DeepSeek-R1 experiences a 22.2% performance drop (Pass@1 from 73.7% to 51.5%) when handling repository-level context compared to previous benchmarks without such context. Meanwhile, we propose a set of more fine-grained evaluation metrics and an enhanced evaluation framework, enabling a more comprehensive analysis of LLMs’ performance in repository-level context code translation tasks to provide fine-grained insights that can effectively inform the development of code translation techniques.
DOI 原文 ·
@inproceedings{paperbot9473,
title = {RustRepoTrans: Repository-level Context Code Translation Benchmark Targeting Rust},
author = {Guangsheng Ou and Mingwei Liu and Yuxuan Chen and Yanlin Wang and Xin Peng and Zibin Zheng},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00057}
}