一句话：提出可扩展多语言框架 Polyglot，基于 IBM CodeNet 从语法、执行、语义与静态度量四维度评测 LLM 的代码翻译质量。
大语言模型在代码相关任务上展现出很大潜力，但其代码翻译能力的可靠性仍缺乏可靠的评估手段。本文提出 Polyglot——一个自动化、多语言、可扩展的框架，用于评测 LLM 在不同编程语言之间的翻译质量。框架以 IBM CodeNet 项目（涵盖多语言的大量编程题目）为数据基础，从语法正确性、执行可靠性、语义保持和静态代码度量四个方面衡量翻译质量。评测聚焦于把 C 代码翻译为 Java、Python 和 Rust，这三种目标语言分属不同范式，也是现代化改造 C 语言系统的常见替代方案。作者使用三种提示策略评测多个开源 LLM，以考察提示方式对翻译表现的影响。结果显示，LLM 在简单代码翻译上表现良好，但在复杂逻辑处理与跨语言范式迁移方面仍有明显局限，需要进一步分析。
Large Language Models (LLMs) show great potential for automating code-related tasks. However, sound assessments are necessary to understand their true capabilities, particularly in code translation, where reliability is crucial. We introduce Polyglot, an automated, multi-language framework for evaluating the translation quality of LLMs between different programming languages. Leveraging the IBM CodeNet Project, an extensive collection of coding problems in multiple languages, we assess translation quality using syntactic correctness, execution reliability, semantic preservation, and static code metrics. Our evaluation focuses on translating C to Java, Python, and Rust, languages that follow distinct paradigms and represent alternatives to modernize C-based systems. We evaluate open-source LLMs using three prompting strategies to understand the impact on translation performance. Our findings highlight that while LLMs show promising results for simple code translation, their limitations regarding complex logic and distinct language paradigms require further analysis.
DOI 原文 ·
@inproceedings{paperbot9556,
title = {Polyglot: An Extensible Framework to Benchmark Code Translation with LLMs},
author = {Marco Vieira and Priyam Ashish Shah and Bhavain Shah and Rrezarta Krasniqi},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00195}
}