一句话：提出神经代码翻译的四级分类体系，并构建覆盖库级与算法级翻译的 G-TransEval 基准，揭示现有评估偏差。
论文关注神经代码翻译评估与基准的不足：现有基准多将源代码视为自然语言，只给出整体准确率，忽略不同翻译类型和复杂度下的模型能力。作者系统考察四个 SOTA 模型，深入分析三个已有基准的优缺点，并基于实证提出四级分类：token 级、语法级、库级和算法级。分析表明，现有模型擅长前两类，但在依赖知识的库级和算法级翻译上表现不佳，且已有基准偏向关键字映射等简单翻译。为此，作者人工整理库级与算法级翻译对及单元测试用例，构建新基准 G-TransEval。结果表明，该基准能更全面、细粒度地刻画代码翻译模型能力，并提供更严格的评估，同时给出构建高阶训练数据和集成多种预训练方法等建议。
In recent years, neural code translation has gained increasing attention. While most of the research focuses on improving model architectures and training processes, we notice that the evaluation process and benchmark for code translation models are severely limited: they primarily treat source code as natural languages and provide a holistic accuracy score while disregarding the full spectrum of model capabilities across different translation types and complexity. In this paper, we present a comprehensive investigation of four state-of-the-art models and analyze in-depth the advantages and limitations of three existing benchmarks. Based on the empirical results, we develop a taxonomy that categorizes code translation tasks into four primary types according to their complexity and knowledge dependence: token level (type 1), syntactic level (type 2), library level (type 3), and algorithm level (type 4). We then conduct a thorough analysis of how existing approaches perform across these four categories. Our findings indicate that while state-of-the-art code translation models excel in type-1 and type-2 translations, they struggle with knowledge-dependent ones such as type-3 and type-4. Existing benchmarks are biased towards trivial translations, such as keyword mapping. To overcome these limitations, we construct G-TransEval, a new benchmark by manually curating type-3 and type-4 translation pairs and unit test cases. Results on our new benchmark suggest that G-TransEval can exhibit more comprehensive and finer-grained capability of code translation models and thus provide a more rigorous evaluation. Our studies also provide more insightful findings and suggestions for future research, such as building type-3 and type-4 training data and ensembling multiple pretraining approaches.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2308-08961,
title = {On the Evaluation of Neural Code Translation: Taxonomy and Benchmark},
author = {Mingsheng Jiao and Tingrui Yu and Xuan Li and Guanjie Qiu and Xiaodong Gu and Beijun Shen},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00114}
}