一句话：系统评估5个神经代码摘要模型在BLEU变体、预处理与数据集上的表现，揭示关键因素并给出评估建议。
本文针对神经代码摘要模型评估中潜在的不一致与偏差问题，系统分析了5个先进模型在6种BLEU变体、4种预处理操作及其组合、3个常用数据集上的表现。研究发现，BLEU变体差异会显著影响结果有效性与模型排名，作者还发现并修复了常用软件包中一个未知的BLEU计算缺陷；人工评估表明BLEU-DC与人类感知最相关。预处理选择可使性能变化-18%至+25%，且预处理组合能提升模型性能。数据集规模、划分方式和重复率等特征也显著影响评估。基于实验结果，作者给出不同场景下的可操作评估建议，并构建共享代码摘要工具箱。
Source code summaries are important for program comprehension and maintenance. However, there are plenty of programs with missing, outdated, or mismatched summaries. Recently, deep learning techniques have been exploited to automatically generate summaries for given code snippets. To achieve a profound understanding of how far we are from solving this problem and provide suggestions to future research, in this paper, we conduct a systematic and in-depth analysis of 5 state-of-the-art neural code summarization models on 6 widely used BLEU variants, 4 pre-processing operations and their combinations, and 3 widely used datasets. The evaluation results show that some important factors have a great influence on the model evaluation, especially on the performance of models and the ranking among the models. However, these factors might be easily overlooked. Specifically, (1) the BLEU metric widely used in existing work of evaluating code summarization models has many variants. Ignoring the differences among these variants could greatly affect the validity of the claimed results. Besides, we discover and resolve an important and previously unknown bug in BLEU calculation in a commonly-used software package. Furthermore, we conduct human evaluations and find that the metric BLEU-DC is most correlated to human perception; (2) code preprocessing choices can have a large (from -18% to +25%) impact on the summarization performance and should not be neglected. We also explore the aggregation of pre-processing combinations and boost the performance of models; (3) some important characteristics of datasets (corpus sizes, data splitting methods, and duplication ratios) have a significant impact on model evaluation. Based on the experimental results, we give actionable suggestions for evaluating code summarization and choosing the best method in different scenarios. We also build a shared code summarization toolbox to facilitate future research.
@inproceedings{paperbot6911,
title = {On the evaluation of neural code summarization},
author = {Ensheng Shi and Yanlin Wang and Lun Du and Junjie Chen and Shi Han and Hongyu Zhang and Dongmei Zhang and Hongbin Sun},
booktitle = {ICSE},
year = {2022},
doi = {10.1145/3510003.3510060}
}