一句话：提出 FreshBrew 基准，系统评估 AI agent 在项目级 Java 迁移中保持语义、避免 reward hacking 的能力。
随着软件生态持续演进，代码库需要不断迁移与现代化，传统做法依赖规则系统与人工介入。LLM 驱动的 agent 框架被视为有潜力的替代方案，但其有效性尚未得到系统评估。本文提出 FreshBrew——一个面向项目级 Java 迁移的 AI agent 基准，重点衡量 agent 保持程序语义、避免 reward hacking 的能力，并因此主张评测应建立在高测试覆盖率的项目之上。该基准包含 228 个代码仓库，作者在其中对比了多种前沿 LLM 与既有规则工具的表现。结果显示，表现最好的模型 Gemini 2.5 Flash 能将 52.3% 的项目成功迁移到 JDK 17。实证分析进一步揭示了当前 agentic 方法的优势、局限以及在真实 Java 现代化任务中出现的典型失败模式，为构建可信的代码迁移系统提供了基础。
AI coding assistants are rapidly becoming integral to modern software development. A key challenge in this space is the continual need to migrate and modernize codebases in response to evolving software ecosystems. Traditionally, such migrations have relied on rule-based systems and human intervention. With the advent of powerful large language models (LLMs), AI-driven agentic frameworks offer a promising alternative—but their effectiveness has not been systematically evaluated. In this paper, we introduce FreshBrew1, a novel benchmark for evaluating AI agents on project-level Java migrations, with a specific focus on measuring an agent’s ability to preserve program semantics and avoid reward hacking, which we argue requires projects with high test coverage for a rigorous and reliable evaluation. We benchmark several state-of-the-art LLMs and compare their performance against established rule-based tools. Our evaluation of AI agents on this benchmark of 228 repositories shows that the top-performing model, Gemini 2.5 Flash, can successfully migrate 52.3% of projects to JDK 17. Our empirical analysis reveals novel insights into the strengths and limitations of current agentic approaches, including characteristic failure modes observed in realistic Java modernization tasks, providing a foundation for evaluating trustworthy code-migration systems. By releasing FreshBrew, we aim to facilitate rigorous, reproducible evaluation and catalyze progress in AI-driven codebase modernization.
@inproceedings{abs-2510-04852,
title = {FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration},
author = {Victor May and Diganta Misra and Yanqi Luo and Anjali Sridhar and Justine Gehring and Silvio Soares Ribeiro},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773214}
}