一句话：构建标准基准与统一工具，在一致配置下实证比较六个SOTA神经程序修复系统，颠覆部分已有结论。
神经程序修复（NPR）系统虽发展迅速，但现有系统在训练数据、评测数据和候选数量等设置上差异巨大，导致比较结论难以公平可信。为缓解这一问题，作者首先构建了一个标准基准数据集与配套框架工具：数据集包含 144,641、13,739 和 13,706 对 Java 缺陷-修复样本，分别用于训练、验证和评测；工具支持选择特定数据集、自动执行 NPR 模型的训练与评测流水线，并通过定义良好的接口便捷集成新模型。基于该基准与工具，作者从可修复性、修复倾向和泛化性三个维度对六个 SOTA NPR 系统进行了全面实证比较。实验结果揭示了这些系统更深层的特性，并推翻了部分已有比较结论，进一步验证了统一实验设置的必要性。此外，作者还总结了 NPR 系统的若干共性（如擅长处理代码删除类缺陷），并据此指出了有前景的研究方向。
Recently, the emerging trend in automatic program repair is to apply deep neural networks to generate fixed code from buggy ones, called NPR (Neural Program Repair). However, the existing NPR systems are trained and evaluated under very different settings (e.g., different training data, inconsistent evaluation data, wide-ranged candidate numbers), which makes it hard to draw fair-enough conclusions when comparing them. Motivated by this, we first build a standard benchmark dataset and an extensive framework tool to mitigate threats for the comparison. The dataset consists of a training set, a validation set and an evaluation set with 144,641, 13,739 and 13,706 bug-fix pairs of Java respectively. The tool supports selecting specific training, validation, and evaluation datasets and automatically conducting the pipeline of training and evaluating NPR models, as well as easily integrating new NPR models by implementing well-defined interfaces. Then, based on the benchmark and tool, we conduct a comprehensive empirical comparison of six SOTA NPR systems w.r.t the repairability, inclination and generalizability. The experimental results reveal deeper characteristics of compared NPR systems and subvert some existing comparative conclusions, which further verify the necessity of unifying the experimental setups in exploring the progresses of NPR systems. Meanwhile, we reveal some common features of NPR systems (e.g., they are good at dealing with code-delete bugs). Finally, we identify some promising research directions derived from our findings.
@inproceedings{paperbot7260,
title = {StandUp4NPR: Standardizing SetUp for Empirically Comparing Neural Program Repair Systems},
author = {Wenkang Zhong and Hongliang Ge and Hongfei Ai and Chuanyi Li and Kui Liu and Jidong Ge and Bin Luo},
booktitle = {ASE},
year = {2022},
doi = {10.1145/3551349.3556943}
}