一句话：提出面向企业代码摘要的聚合评估框架，强调独特性和完整性，惩罚冗长与冗余，并提升与人工评分的一致性。
企业代码库复杂且规模大，现有公开代码数据集与评估方法多针对小型简单代码片段，难以处理企业代码摘要中的重复、冗长和不完整问题；正确性之外，独特性和完整性等关键维度也研究不足。本文提出一种面向企业需求的新评估框架，采用聚合指标，优先衡量独特性和完整性，惩罚冗长与冗余，奖励信息量及与底层代码的一致性。作者在人工标注的企业 Java 数据集上进行初步实验，相较基线将 RMSE 改善 7.4%；其独特性和完整性指标与人工评分的相关性分别提升 32% 和 5.3%。结果表明该框架更适合企业代码摘要质量评估。
Evaluating the quality of code summaries is essential for enterprise software, where the complexity and scale of codebases introduce unique challenges that are inadequately addressed by existing public code datasets and evaluation methods. These methods, typically designed for small and straightforward code snippets, often overlook critical issues such as repetitiveness, verbosity, and incompleteness—issues that are particularly prominent in enterprise-level code summaries. While correctness has been extensively studied, other dimensions critical to enterprise contexts, such as distinctiveness and completeness, remain underexplored. To address these gaps, we propose a novel evaluation framework that emphasizes aggregated metrics tailored to enterprise needs, prioritizing both distinctiveness and completeness. This framework introduces metrics designed to penalize verbosity and redundancy while rewarding informativeness and alignment with the underlying code. Initial experiments conducted on human-annotated enterprise Java datasets demonstrate the effectiveness of our approach by improving the RMSE values by 7.4% over the baselines. Correlation studies of our distinctiveness and completeness metrics with human ratings also shows improvement of 32% and 5.3% respectively over the baselines.
@inproceedings{paperbot9086,
title = {Measuring What Matters: An Aggregate Metric for Assessing Enterprise Code Summaries},
author = {Ashita Saxena and Palanivel Kodeswaran and Sayandeep Sen and Srikanth Tamilselvam},
booktitle = {FSE},
year = {2025},
doi = {10.1145/3696630.3728520}
}