一句话：从 GitHub 真实 LLM 使用场景出发构建多语言代码生成基准，揭示模型与现有基准间的差距。
现有代码生成基准（如 CoderEval、ClassEval）任务过难，GPT-4 在 ClassEval 上 pass@1 仅 37.0%，而开发者实际只把 LLM 用于其能可靠完成的任务。作者从 GitHub 挖掘标注为 ChatGPT/Copilot 生成的代码，整理出真实编程任务，改写并生成参考解与测试用例，再翻译到多种语言，构建含 376 题的 RealisticCodeBench。评测 12 个通用与代码专用 LLM 后：GPT-4.1 多语言平均 pass@1 最高，DeepSeek-V3-671B 紧随其后，CodeGeeX4-9B 可作 GPT-4o-mini 的廉价替代；与 HumanEval 的差异说明部分模型过度特化，文中还总结了失败模式与启示。
Evaluating the code generation capabilities of Large Language Models (LLMs) remains an open question. Recently, more advanced benchmarks—such as CoderEval, EvoCodeBench, and ClassEval—have been introduced to evaluate LLMs on practical coding tasks from GitHub repositories, such as non-standalone function generation and class-level code generation. However, even the most sophisticated LLMs struggle with these complex tasks; for instance, GPT-4 achieves only a 37.0% pass@1 on ClassEval. Prior studies show that developers often discard LLM-generated code or abandon code generation models when outputs are incorrect or require extensive debugging, which leads them to rely on LLMs primarily for code generation tasks that high-performing models can reliably handle.In response to this gap, we introduce RealisticCodeBench, a benchmark specifically designed to reflect the types of problems developers commonly tackle with LLMs. By mining GitHub repositories for code samples tagged as generated by ChatGPT or Copilot, we collect real-world coding tasks that capture typical LLM usage scenarios. We modify these tasks, generate reference solutions and test cases, and adapt the problems into multiple programming languages. This effort results in RealisticCodeBench, comprising a total of 376 programming problems translated across multiple languages: 361 in Python, 346 in JavaScript, 343 in TypeScript, 307 in Java, and 323 in C++, each with corresponding reference solutions and test cases. We evaluate 12 general-purpose and code-specific LLMs on RealisticCodeBench. Our findings reveal that GPT-4.1 achieves the highest average pass@1 score across languages, closely followed by DeepSeek-V3-671B, suggesting that DeepSeek-V3-671B provides a viable open-source alternative to GPT-4.1 for large companies with sufficient GPU resources and privacy concerns. CodeGeeX4-9B, a cost-effective model, emerges as a suitable substitute for GPT-4o-mini for individual developers and smaller organizations with similar privacy considerations. Additionally, LLM performance discrepancies between HumanEval and RealisticCodeBench suggest that some LLMs are either overly specialized for HumanEval-style problems or insufficiently optimized for real-world coding challenges. Finally, we analyze failed cases, summarize common LLM limitations, and provide implications for researchers and practitioners.
DOI 原文 ·
@inproceedings{paperbot9410,
title = {RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code Generation},
author = {Xiao Yu and Haoxuan Chen and Lei Liu and Xing Hu and Jacky Wai Keung and Xin Xia},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00248}
}