一句话：本文系统评估代码摘要基准数据集的数据预处理噪声，提出自动清洗工具，证明去噪能显著提升模型性能。
代码摘要模型通常在常用基准数据集上训练和验证，但这些从真实项目构建的数据集质量鲜有审视。本文针对四个广泛使用的代码摘要基准数据集，提出自动化代码-注释清洗工具，能够检测因不当数据预处理操作产生的噪声数据；随后用该工具评估四个数据集的数据质量，并通过对比实验考察噪声对摘要模型性能的影响。结果显示，数据预处理噪声普遍存在于所有四个基准数据集中，去除这些噪声数据可显著提升代码摘要性能。作者认为这些发现有助于理解代码摘要任务中的数据质量问题，并为后续研究与实践提供参考。
Code summarization, the task of generating useful comments given the code, has long been of interest. Most of the existing code summarization models are trained and validated on widely-used code comment benchmark datasets. However, little is known about the quality of the benchmark datasets built from real-world projects. Are the benchmark datasets as good as expected? To bridge the gap, we conduct a systematic research to assess and improve the quality of four benchmark datasets widely used for code summarization tasks. First, we propose an automated code-comment cleaning tool that can accurately detect noisy data caused by inappropriate data preprocessing operations from existing benchmark datasets. Then, we apply the tool to further assess the data quality of the four benchmark datasets, based on the detected noises. Finally, we conduct comparative experiments to investigate the impact of noisy data on the performance of code summarization models. The results show that these data preprocessing noises widely exist in all four benchmark datasets, and removing these noisy data leads to a significant improvement on the performance of code summarization. We believe that the findings and insights will enable a better understanding of data quality in code summarization tasks, and pave the way for relevant research and practice.
@inproceedings{paperbot7111,
title = {Are we building on the rock? on the importance of data preprocessing for code summarization},
author = {Lin Shi and Fangwen Mu and Xiao Chen and Song Wang and Junjie Wang and Ye Yang and Ge Li and Xin Xia and Qing Wang},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549145}
}