一句话：指出现有代码生成评估多停留在函数/语句级，并强调需在类级场景考察LLM能力。
本文关注大型语言模型（LLM）在代码生成中的评估问题。现有基准如HumanEval多要求模型根据自然语言描述生成单个函数或语句等独立、小规模代码单元，虽便于比较模型，却难以反映真实软件开发中的类级代码生成能力。作者指出这种函数级/语句级评估与现实开发场景存在差距，因此题目聚焦类级代码生成评估，试图回答LLM在更接近真实软件开发的类级任务中表现如何。不过，所给摘要主要说明研究动机与现有评估的局限，未详述具体方法、基准构建或实验结果。
Recently, many large language models (LLMs) have been proposed, showing advanced proficiency in code generation. Meanwhile, many efforts have been dedicated to evaluating LLMs on code generation benchmarks such as HumanEval. Although being very helpful for comparing different LLMs, existing evaluation focuses on a simple code generation scenario (i.e., function-level or statement-level code generation), which mainly asks LLMs to generate one single code unit (e.g., a function or a statement) for the given natural language description. Such evaluation focuses on generating independent and often small-scale code units, thus leaving it unclear how LLMs perform in real-world software development scenarios.
@inproceedings{paperbot11412,
title = {Evaluating Large Language Models in Class-Level Code Generation},
author = {Xueying Du and Mingwei Liu and Kaixin Wang and Hanlin Wang and Junwei Liu and Yixuan Chen and Jiayi Feng and Chaofeng Sha and Xin Peng and Yiling Lou},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639219}
}