一句话：构建迄今最大的人工标注补丁正确性检查数据集 PraPatch，重新评估九种最先进 PCC 技术并给出改进方向。
自动程序修复（APR）生成的补丁即便通过全部原始测试也未必正确，即“可信补丁问题”；补丁正确性检查（PCC）技术应运而生，但现有研究仅在有限数据集上评估，外部有效性存疑。本文构建了迄今最大的人工标注 PCC 数据集 PraPatch，含 PraPR（字节码级、可在大型预定义搜索空间内穷举可信补丁）生成的 1,988 个补丁，并重新评估九种最先进 PCC 技术。结果发现：静态技术“正确补丁更接近缺陷代码”的假设不再成立；基于学习的技术易过拟合数据集；动态技术整体仍有效，但在改动更复杂的补丁上性能明显下降；基于自然性的技术显著优于传统静态技术，是有前景的方向。
Automated Program Repair (APR) techniques have drawn wide attention from both academia and industry. Meanwhile, one main limitation with the current state-of-the-art APR tools is that patches passing all the original tests are not necessarily the correct ones wanted by developers, i.e., the plausible patch problem. To date, various Patch-Correctness Checking (PCC) techniques have been proposed to address this important issue. However, they are only evaluated on very limited datasets as the APR tools used for generating such patches can only explore a small subset of the search space of possible patches, posing serious threats to external validity to existing PCC studies. In this paper, we construct an extensive PCC dataset, PraPatch (the largest manually labeled PCC dataset to our knowledge), to revisit all nine state-of-the-art PCC techniques. More specifically, our PCC dataset PraPatch includes 1,988 patches generated from the recent PraPR APR tool, which leverages highly-optimized bytecode-level patch executions and can exhaustively explore all possible plausible patches within its large predefined search space (including well-known fixing patterns from various prior APR tools). Our extensive study of representative PCC techniques on PraPatch has revealed various findings, including: 1) the assumption made by existing static PCC techniques that correct patches are more similar to buggy code than incorrect plausible patches no longer holds, 2) state-of-the-art learning-based techniques tend to suffer from the dataset overfitting problem, 3) while dynamic techniques overall retain their effectiveness on our new dataset, their performance drops substantially on patches with more complicated changes and 4) the very recent naturalness-based techniques can substantially outperform traditional static techniques and could be a promising direction for PCC. Based on our findings, we also provide various guidelines/suggestions for advancing PCC in the near future.
@inproceedings{YangWL0023,
title = {A Large-Scale Empirical Review of Patch Correctness Checking Approaches},
author = {Jun Yang and Yuehan Wang and Yiling Lou and Ming Wen and Lingming Zhang},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616331}
}