一句话：通过大规模实证研究系统比较现有自动补丁正确性评估技术，并提出学习与投票结合的集成策略以显著提升过拟合补丁检测效果。
基于测试的自动程序修复虽进展显著，但生成的补丁常因过拟合而仅“看似正确”。现有补丁正确性评估技术缺乏系统比较，优劣尚不明确。本文对静态与动态评估技术开展大规模实证研究，基于 21 个 APR 工具、4 类方法生成的 902 个补丁进行评测。结果发现：补丁语法与语义的静态特征能有效区分过拟合补丁；动态技术精度高，静态启发式召回更优；技术效果受项目和 APR 类型影响；各技术高度互补，单技术最多检出 53.5% 的过拟合补丁，而 oracle 下至少一种技术可检出 93.3%。据此提出先通过学习集成静态特征、再用多数投票结合其他技术的集成策略，实验表明可显著提升现有评估技术性能。
Test-based automated program repair (APR) has attracted huge attention from both industry and academia. Despite the significant progress made in recent studies, the overfitting problem (i.e., the generated patch is plausible but overfitting) is still a major and long-standing challenge. Therefore, plenty of techniques have been proposed to assess the correctness of patches either in the patch generation phase or in the evaluation of APR techniques. However, the effectiveness of existing techniques has not been systematically compared and little is known to their advantages and disadvantages. To fill this gap, we performed a large-scale empirical study in this paper. Specifically, we systematically investigated the effectiveness of existing automated patch correctness assessment techniques, including both static and dynamic ones, based on 902 patches automatically generated by 21 APR tools from 4 different categories. Our empirical study revealed the following major findings: (1) static code features with respect to patch syntax and semantics are generally effective in differentiating overfitting patches over correct ones; (2) dynamic techniques can generally achieve high precision while heuristics based on static code features are more effective towards recall; (3) existing techniques are more effective towards certain projects and types of APR techniques while less effective to the others; (4) existing techniques are highly complementary to each other. For instance, a single technique can only detect at most 53.5% of the overfitting patches while 93.3% of them can be detected by at least one technique when the oracle information is available. Based on our findings, we designed an integration strategy to first integrate static code features via learning, and then combine with others by the majority voting strategy. Our experiments show that the strategy can enhance the performance of existing patch correctness assessment techniques significantly.
DOI 原文 ·
@inproceedings{paperbot5973,
title = {Automated patch correctness assessment},
author = {Shangwen Wang and Ming Wen and Bo Lin and Hongjun Wu and Yihao Qin and Deqing Zou and Xiaoguang Mao and Hai Jin},
booktitle = {ASE},
year = {2020},
doi = {10.1145/3324884.3416590}
}