一句话：提出可执行、统一的 Java 基准 CoderUJB，含 2239 道题覆盖五类实际编程任务，系统评测大语言模型的编码能力。
针对现有代码基准过于简化、难以反映真实软件开发多任务特性的问题，本文提出 CoderUJB——一个可执行且贴近实际开发场景的 Java 编程基准。该基准从 17 个真实开源 Java 项目中提取 2239 道编程题，覆盖五类实际编程任务。作者在基准上对多种开源与闭源大语言模型进行实证研究，考察特定编程语言继续预训练与指令微调对模型表现的影响。结果显示，大语言模型虽具较强潜力，但在非功能性代码生成（如测试生成与缺陷检测）上仍面临挑战；同时，特定语言的继续预训练与指令微调有时反而会损害某些任务上的性能，提示需要更细致的训练策略。该工作为更真实地评估大语言模型的编程能力提供了重要基准与实证洞见。
In the evolving landscape of large language models (LLMs) tailored for software engineering, the need for benchmarks that accurately reflect real-world development scenarios is paramount. Current benchmarks are either too simplistic or fail to capture the multi-tasking nature of software development. To address this, we introduce CoderUJB, a new benchmark designed to evaluate LLMs across diverse Java programming tasks that are executable and reflective of actual development scenarios, acknowledging Java's prevalence in real-world software production. CoderUJB comprises 2,239 programming questions derived from 17 real open-source Java projects and spans five practical programming tasks. Our empirical study on this benchmark investigates the coding abilities of various open-source and closed-source LLMs, examining the effects of continued pre-training in specific programming languages code and instruction fine-tuning on their performance. The findings indicate that while LLMs exhibit strong potential, challenges remain, particularly in non-functional code generation (e.g., test generation and defect detection). Importantly, our results advise caution in the specific programming languages continued pre-training and instruction fine-tuning, as these techniques could hinder model performance on certain tasks, suggesting the need for more nuanced strategies. CoderUJB thus marks a significant step towards more realistic evaluations of programming capabilities in LLMs, and our study provides valuable insights for the future development of these models in software engineering.
DOI 原文 ·
@inproceedings{paperbot8567,
title = {CoderUJB: An Executable and Unified Java Benchmark for Practical Programming Scenarios},
author = {Zhengran Zeng and Yidong Wang and Rui Xie and Wei Ye and Shikun Zhang},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3652115}
}