一句话：大规模实证研究揭示LLM代码翻译正确率有限并归纳15类翻译缺陷，提出基于错误症状的提示改进方法。
代码翻译旨在将源代码从一种编程语言转换为另一种。鉴于大语言模型（LLM）在代码合成上的潜力，研究者开始探索其自动化代码翻译的能力。本文开展大规模实证研究，考察通用LLM与代码LLM在C、C++、Go、Java、Python等语言对之间的翻译表现。研究涵盖来自三个基准和两个真实项目的1700个代码样本，发现LLM尚不能可靠地自动化代码翻译，正确翻译率仅为2.1%至47.3%。通过对失败翻译的人工分析，作者归纳出15类翻译缺陷，并将LLM方法与传统的非LLM方法进行比较，发现两类技术各有优劣。基于错误翻译的症状，作者提出一种提示构造方法，使LLM代码翻译性能平均提升5.5%。研究还发布了包含1700个样本、5种语言、10K+测试、43K+翻译代码、1748个人工标注缺陷和1365个缺陷修复对的数据集，以推动该领域研究。
Code translation aims to convert source code from one programming language (PL) to another. Given the promising abilities of large language models (LLMs) in code synthesis, researchers are exploring their potential to automate code translation. The prerequisite for advancing the state of LLM-based code translation is to understand their promises and limitations over existing techniques. To that end, we present a large-scale empirical study to investigate the ability of general LLMs and code LLMs for code translation across pairs of different languages, including C, C++, Go, Java, and Python. Our study, which involves the translation of 1,700 code samples from three benchmarks and two real-world projects, reveals that LLMs are yet to be reliably used to automate code translation---with correct translations ranging from 2.1% to 47.3% for the studied LLMs. Further manual investigation of unsuccessful translations identifies 15 categories of translation bugs. We also compare LLM-based code translation with traditional non-LLM-based approaches. Our analysis shows that these two classes of techniques have their own strengths and weaknesses. Finally, insights from our study suggest that providing more context to LLMs during translation can help them produce better results. To that end, we propose a prompt-crafting approach based on the symptoms of erroneous translations; this improves the performance of LLM-based code translation by 5.5% on average. Our study is the first of its kind, in terms of scale and breadth, that provides insights into the current limitations of LLMs in code translation and opportunities for improving them. Our dataset---consisting of 1,700 code samples in five PLs with 10K+ tests, 43K+ translated code, 1,748 manually labeled bugs, and 1,365 bug-fix pairs---can help drive research in this area.
@inproceedings{paperbot11385,
title = {Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code},
author = {Rangeet Pan and Ali Reza Ibrahimzada and Rahul Krishna and Divya Sankar and Lambert Pouguem Wassi and Michele Merler and Boris Sobolev and Raju Pavuluri and Saurabh Sinha and Reyhaneh Jabbarvand},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639226}
}