一句话：提出演化感知的仓库级代码生成基准 HumanEvo，揭示忽略软件演化的评测会高估 LLM 性能 10.0%–61.1%。
现有仓库级代码生成评测通常只使用项目最新版本的上下文，忽略软件项目随时间的演化，导致对 LLM 性能的评估失真。本文构建演化感知的仓库级代码生成数据集 HumanEvo，并提供基于自动执行的评估工具；同时按依赖级别对样本进行人工分类，以分析不同依赖程度函数的生成表现。作者在 7 个代表性 LLM 上开展实验，发现忽略演化的评测会高估模型性能，不同上下文获取方法下高估幅度为 10.0% 至 61.1%。基于这些发现，论文给出了更真实评估 LLM 代码生成的可操作建议，并发布了共享的演化感知代码生成工具箱。
To evaluate the repository-level code generation capabilities of Large Language Models (LLMs) in complex real-world software development scenarios, many evaluation methods have been developed. These methods typically leverage contextual code from the latest version of a project to assist LLMs in accurately generating the desired function. However, such evaluation methods fail to consider the dynamic evolution of software projects over time, which we refer to as evolution-ignored settings. This in turn results in inaccurate evaluation of LLMs' performance. In this paper, we conduct an empirical study to deeply understand LLMs' code generation performance within settings that reflect the evolution nature of software development. To achieve this, we first construct an evolution-aware repository-level code generation dataset, namely HumanEvo, equipped with an automated execution-based evaluation tool. Second, we manually categorize HumanEvo according to dependency levels to more comprehensively analyze the model's performance in generating functions with different dependency levels. Third, we conduct extensive experiments on HumanEvo with seven representative and diverse LLMs to verify the effectiveness of the proposed benchmark. We obtain several important findings through our experimental study. For example, we find that previous evolution-ignored evaluation methods result in inflated performance of LLMs, with performance overestimations ranging from 10.0% to 61.1% under different context acquisition methods, compared to the evolution-aware evaluation approach. Based on the findings, we give actionable suggestions for more realistic evaluation of LLMs on code generation. We also build a shared evolution-aware code generation toolbox to facilitate future research. The replication package including source code and datasets is anonymously available at https://github.Com/DeepSoftwareAnalytics/HumanEvo.
DOI 原文 ·
@inproceedings{paperbot8867,
title = {HumanEvo: An Evolution-Aware Benchmark for More Realistic Evaluation of Repository-Level Code Generation},
author = {Dewu Zheng and Yanlin Wang and Ensheng Shi and Ruikai Zhang and Yuchi Ma and Hongyu Zhang and Zibin Zheng},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00228}
}