一句话：在统一环境下多维度评测12种APR技术，揭示基于LLM方法的优势、可编译性短板与传统技术的互补性。
由于缺陷在真实程序中不可避免，自动程序修复（APR）技术不断涌现，但现有评测缺乏统一标准，设置与基准各异，且通常仅采用合理性（plausibility）与真实性（genuineness）指标，可能掩盖技术中的潜在问题。本文在相同环境与设置下，对9种基于学习与3种传统的最先进APR技术开展广泛的多维度评测。评测采用广泛使用的Defects4J V2.0.0基准，以及基于变异构造的大规模基准MuBench（含1700个由不同变异算子生成的人工缺陷），并运用可编译性/合理性/真实性指标与SYE、TCE指标，深入分析1,814,652个生成补丁。研究发现：基于大语言模型（LLM）的APR在Defects4J V1.2.0上过拟合程度更低且修复缺陷最多；传统与学习型技术具有互补优势，二者结合前景可期；学习型技术的补丁可编译性仍待提升；此外还揭示了处理不可解析符号编译问题、减少重复/无效补丁等改进方向，并发现7个实现问题，其中5个已被原作者确认并修复。
As bugs are inevitable and prevalent in real-world programs, many Automated Program Repair (APR) techniques have been proposed to generate patches for them. However, due to the lack of a standard for evaluating APR techniques, prior works tend to use different settings and benchmarks in evaluation, threatening the trustworthiness of the evaluation results. Additionally, they typically only adopt plausibility and genuineness as evaluation metrics, which may potentially mask some underlying issues in APR techniques. To overcome these issues, in this paper, we conduct an extensive and multi-dimensional evaluation of nine learning-based and three traditional state-of-the-art APR techniques under the same environment and settings. We employ the widely studied Defects4J V2.0.0 benchmark and a newly constructed large-scale mutation-based benchmark named MuBench, derived from Defects4J and including 1,700 artificial bugs generated by various mutators, to uncover potential limitations in these APR techniques. We also apply multi-dimensional metrics, including compilability/plausibility/genuineness metrics, as well as SYE (SYntactic Equivalence) and TCE (Trivial Compiler Equivalence) metrics, to thoroughly analyze the 1,814,652 generated patches. This paper presents noteworthy findings from the extensive evaluation: Firstly, Large Language Model (LLM) based APR demonstrates less susceptibility to overfitting on the Defects4J V1.2.0 dataset and fixes the most number of bugs. Secondly, the study suggests a promising future for combining traditional and learning-based APR techniques, as they exhibit complementary advantages in fixing different types of bugs. Additionally, this work highlights the necessity for further enhancing patch compilability of learning-based APR techniques, despite the presence of various existing strategies attempting to improve it. The study also reveals other guidelines for enhancing APR techniques, including the need for handling unresolvable symbol compilability issues and reducing duplicate/no-op patch generation. Finally, our study uncovers seven implementation issues in the studied techniques, with five of them confirmed and fixed by the corresponding authors.
DOI 原文 ·
@inproceedings{paperbot8458,
title = {Benchmarking Automated Program Repair: An Extensive Study on Both Real-World and Artificial Bugs},
author = {Yicheng Ouyang and Jun Yang and Lingming Zhang},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3652140}
}