一句话：提出 ComplexCodeEval 基准，用更复杂代码和多任务场景评估大代码模型，弥补现有基准场景单一的问题。
现有面向大代码模型（LCM）的评测基准往往只覆盖单一场景（如代码生成或补全），且题目未必来自真实开发实践，难以反映 LCM 在多样、复杂开发任务中的表现。为此，本文提出 ComplexCodeEval：一个面向更复杂代码、覆盖代码生成、代码补全、API 推荐与测试函数生成等多类任务的评测基准。其目标是把评测从单一任务扩展到贴近真实开发流程的多任务情境，并关注开发者实际面临的编程挑战。该基准可用于系统比较不同 LCM 在复杂代码上的能力，揭示现有基准可能遗漏的薄弱环节。
In recent years, with the widespread attention of academia and industry on the application of large language models (LLMs) to code-related tasks, an increasing number of large code models (LCMs) have been proposed and corresponding evaluation benchmarks have continually emerged. Although existing evaluation benchmarks are helpful for comparing different LCMs, they may not reflect the performance of LCMs in various development scenarios. Specifically, they might evaluate model performance in only one type of scenario (e.g., code generation or code completion), whereas real development contexts are diverse and may involve multiple tasks such as code generation, code completion, API recommendation, and test function generation. Additionally, the questions may not originate from actual development practices, failing to capture the programming challenges faced by developers during the development process.
@inproceedings{paperbot8444,
title = {ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code},
author = {Jia Feng and Jiachen Liu and Cuiyun Gao and Chun Yong Chong and Chaozheng Wang and Shan Gao and Xin Xia},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695552}
}