一句话：提出首个面向软件工程生成物正确性评估的 LLM 集成评审度量，显著提升与人工判断的相关性。
大语言模型等自动化技术被广泛用于生成代码片段、补丁和注释等软件工件，但如何准确评估其正确性仍是难题：人工评估准确却难以扩展，自动度量可扩展却常无法反映真实正确性。本文提出 SE-Jury，首个专为准确评估生成软件工件正确性而设计的 LLM 集成评审度量。它先定义五种不同的评估策略，每种实现为独立评审者；再通过动态团队选择机制，挑出最合适的评审者子集组成团队，以集成方式产生最终正确性分数。作者在代码生成、自动程序修复和代码摘要三类软件工程任务的多个基准上评估 SE-Jury。结果表明，SE-Jury 与人工判断的相关性持续更高，相较现有自动度量提升 29.6%–140.8%；在代码生成和程序修复任务中，其与人工标注者的一致程度接近标注者间一致性，显示出作为可扩展且可靠的人工评估替代方案的潜力。
Large Language Models (LLMs) and other automated techniques have been increasingly used to support software developers by generating software artifacts such as code snippets, patches, and comments. However, accurately assessing the correctness of these generated artifacts remains a significant challenge. On one hand, human evaluation provides high accuracy but is labor-intensive and lacks scalability. On the other hand, many automatic evaluation metrics are scalable and require minimal human effort, but they often fail to accurately reflect the actual correctness of generated software artifacts.In this paper, we present SE-Jury, the first evaluation metric for LLM-as-Ensemble-Judge specifically designed to accurately assess the correctness of generated software artifacts. SE-Jury first defines five distinct evaluation strategies, each implemented as an independent judge. A dynamic team selection mechanism then identifies the most appropriate subset of judges as a team to produce a final correctness score through ensembling. We evaluate SE-Jury across a diverse set of software engineering (SE) benchmarks that span three popular SE tasks: code generation, automated program repair, and code summarization. Results demonstrate that SE-Jury consistently achieves a higher correlation with human judgments, with improvements ranging from 29.6%–140.8% over existing automatic metrics. SE-Jury reaches agreement levels with human annotators that are close to inter-annotator agreement in code generation and program repair. These findings underscore SE-Jury’s potential as a scalable and reliable alternative to human evaluation in these SE tasks.
DOI 原文 ·
@inproceedings{paperbot9302,
title = {SE-Jury: An LLM-as-Ensemble-Judge Metric for Narrowing the Gap with Human Evaluation in SE},
author = {Xin Zhou and Kisub Kim and Ting Zhang and Martin Weyssow and Luís F. Gomes and Guang Yang and Kui Liu and Xin Xia and David Lo},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00214}
}