一句话：提出首个基于智能体的复杂代码生成评估框架 CodeVisionary，通过需求分解与多评审协商实现细粒度、可解释评估。
现有 LLM 代码生成评估可分为人工、指标和 LLM 三类；LLM 方法虽具上下文理解优势，但多依赖静态提示，难以处理多需求、复杂上下文的代码场景，且缺乏细粒度与可解释性。本文提出 CodeVisionary，首个基于智能体的复杂代码生成评估框架，分两阶段：需求引导的多维上下文蒸馏将任务需求分解并逐步收集上下文；细粒度评分与总结通过自我导向和协商动作，让多个评审从多视角理解代码并讨论达成共识，并生成可解释报告。作者构建含 363 样本、覆盖 37 种编码场景和 23 种编程语言的基准。实验显示其优于三个基线，在 Pearson、Spearman、Kendall-Tau 上平均提升 0.217、0.163、0.141。
Large language models (LLMs) have demonstrated strong capabilities in code generation, underscoring the critical need for rigorous and comprehensive evaluation. Existing evaluation approaches fall into three categories, including human-centered, metric-based, and LLM-based. Considering that human-centered approaches are labour-intensive and metric-based ones overly rely on reference answers, LLM-based approaches are gaining increasing attention due to their stronger contextual understanding capabilities. However, they generally evaluate the generated code based on static prompts, and tend to fail for complex code scenarios which typically involve multiple requirements and require more contextual information. In addition, these approaches lack fine-grained evaluation for complex code, resulting in limited explainability.To mitigate the limitations, we propose CodeVisionary, the first agent-based evaluation framework for complex code generation. CodeVisionary consists of two stages: (1) Requirement-guided multi-dimensional context distillation stage, which first formulates a detailed evaluation plan by decomposing task requirements, and then stepwise collects multi-dimensional contextual information for each requirement. (2) Fine-grained scoring and summarization stage, which defines self-directed and negotiation-based actions, allowing multiple judges to comprehend complex code from fine-grained and diverse viewpoints, and reach a consensus through discussion. A comprehensive evaluation report is also generated for enhanced explainability. For validation, we construct a new benchmark consisting of 363 samples spanning 37 coding scenarios and 23 programming languages. Extensive experiments demonstrate that CodeVisionary achieves the best performance among three baselines for evaluating complex code generation, outperforming the best baseline with average improvements of 0.217, 0.163, and 0.141 in Pearson, Spearman, and Kendall-Tau coefficients, respectively. The resources of CodeVisionary are available at https://github.com/Eshe0922/CodeVisionary.
DOI 原文 ·
@inproceedings{paperbot9576,
title = {An Agent-based Evaluation Framework for Complex Code Generation},
author = {Xinchen Wang and Ruida Hu and Pengfei Gao and Chao Peng and Cuiyun Gao},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00200}
}