一句话：将补丁正确性评估转化为问答问题，用 bug 报告与补丁描述的语义相关性预测 APR 补丁是否正确。
自动程序修复（APR）工具容易生成过拟合补丁，而测试套件等 oracle 难以可靠判定补丁正确性。现有工作多依赖工程化/深度特征或动态执行信息。本文提出新视角：正确的补丁应当“回答”由缺陷行为提出的问题，因此将补丁正确性评估建模为问答（QA）问题，利用自然语言处理模型评估缺陷（问题）与补丁（答案）之间的语义相关性。所提方法 Quatrain 以 bug 报告和生成补丁的自然语言描述为输入，先用当前最优的 commit message 生成模型为每个补丁产生相关描述，再通过神经网络学习 bug 报告与 commit message 的语义关联。在 Defects4j、Bugs.jar 和 Bears 三个缺陷数据集上生成的 9135 个补丁上进行实验，Quatrain 在预测补丁正确性上达到 AUC 0.886，召回 93% 的正确补丁并过滤掉 62% 的错误补丁。实验还表明输入质量会影响预测性能，并验证模型确实学到了 bug 报告与代码改动描述之间的关系；最后与先前工作比较并讨论其优势。
Patch correctness has been the focus of automated program repair (APR) in recent years due to the propensity of APR tools to generate overfitting patches. Given a generated patch, the oracle (e.g., test suites) is generally weak in establishing correctness. Therefore, the literature has proposed various approaches of leveraging machine learning with engineered and deep learned features, or exploring dynamic execution information, to further explore the correctness of APR-generated patches. In this work, we propose a novel perspective to the problem of patch correctness assessment: a correct patch implements changes that “answer” to a problem posed by buggy behavior. Concretely, we turn the patch correctness assessment into a Question Answering problem. To tackle this problem, our intuition is that natural language processing can provide the necessary representations and models for assessing the semantic correlation between a bug (question) and a patch (answer). Specifically, we consider as inputs the bug reports as well as the natural language description of the generated patches. Our approach, Quatrain, first considers state-of-the-art commit message generation models to produce the relevant inputs associated to each generated patch. Then we leverage a neural network architecture to learn the semantic correlation between bug reports and commit messages. Experiments on a large dataset of 9 135 patches generated for three bug datasets (Defects4j, Bugs.jar and Bears) show that Quatrain achieves an AUC of 0.886 on predicting patch correctness, and recalling 93% correct patches while filtering out 62% incorrect patches. Our experimental results further demonstrate the influence of inputs quality on prediction performance. We further perform experiments to highlight that the model indeed learns the relationship between bug reports and code change descriptions for the prediction. Finally, we compare against prior work and discuss the benefits of our approach.
@inproceedings{paperbot7282,
title = {Is this Change the Answer to that Problem?},
author = {Haoye Tian and Xunzhu Tang and Andrew Habib and Shangwen Wang and Kui Liu and Xin Xia and Jacques Klein and TegawendÉ F. BissyandÉ},
booktitle = {ASE},
year = {2022},
doi = {10.1145/3551349.3556914}
}