一句话：提出 CoderEval 基准，评估生成式预训练模型在真实代码场景下的代码生成能力。
基于预训练与微调的代码生成模型（如 Codex、CodeGen、PanGu-Coder）日益受到关注，但现有基准（如 HumanEval、AiXBench）只包含独立函数生成，即函数仅调用内置函数或标准库。作者指出，非独立函数在流行开源项目中占比超过 70%，仅评估独立函数无法反映模型在开源或专有代码等实用场景中的代码生成效果。为此，论文提出 CoderEval 基准，面向实用代码生成，覆盖更贴近真实工程的函数生成任务，以支持对生成式预训练模型有效性的系统评估。
Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks (e.g., HumanEval and AiXBench) are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios (i.e., code generation for real settings of open source or proprietary code).
DOI 原文 ·
@inproceedings{paperbot11281,
title = {CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models},
author = {Hao Yu and Bo Shen and Dezhi Ran and Jiaxin Zhang and Qi Zhang and Yuchi Ma and Guangtai Liang and Ying Li and Qianxiang Wang and Tao Xie},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3623316}
}