一句话：提出首个面向 C/C++ 的可执行程序修复基准 Defects4C，并据此实证评测 24 个先进大模型的修复能力。
自动程序修复（APR）对提升软件质量与可靠性至关重要，但由于长期缺乏高质量、开源的 C/C++ 基准，相关研究远落后于受益于 Defects4J 等基准的 Java 领域，而 C/C++ 使用广泛且漏洞频发。为填补这一空白，作者提出 Defects4C：一个面向 C/C++ 程序修复的综合性可执行基准。该数据集取自真实世界 C/C++ 仓库，包含约 900 万条与缺陷相关的提交、248 个高质量缺陷函数与 102 个易受攻击函数，并为每个样本配套可复现的测试用例，既支持对修复技术的严格评测，也支持对基于学习的方法进行再训练。基于该基准，作者对 24 个当前最先进的大语言模型开展了系统的实证研究，揭示了大模型在 C/C++ 修复任务上的优势与局限，说明仍需更鲁棒的方法，并强调 Defects4C 对推动后续研究的价值。
Automated Program Repair (APR) plays a critical role in enhancing the quality and reliability of software systems. While substantial progress has been made in Java-based APR, largely facilitated by benchmarks like Defects4J, there remains a significant gap in research on C/C++ program repair, despite the widespread use of C/C++ and the prevalence of associated vulnerabilities. This gap is primarily due to the lack of high-quality, open-source benchmarks tailored for C/C++.To address this issue, we introduce Defects4C, a comprehensive and executable benchmark specifically designed for C/C++ program repair. Our dataset is constructed from real-world C/C++ repositories and includes a large collection of bug-relevant commits (9M in total), 248 high-quality buggy functions, and 102 vulnerable functions, all paired with test cases for reproduction. These resources enable rigorous evaluation of repair techniques and support the retraining of learning-based approaches for enhanced performance.Using Defects4C, we conduct a comprehensive empirical study evaluating the effectiveness of 24 state-of-the-art large language models (LLMs) in repairing C/C++ faults. Our findings offer valuable insights into the strengths and limitations of current LLM-based APR techniques in this domain, highlighting both the need for more robust methods and the critical role of Defects4C in advancing future research.
DOI 原文 ·
@inproceedings{paperbot9292,
title = {Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs},
author = {Jian Wang and Xiaofei Xie and Qiang Hu and Shangqing Liu and Jiongchi Yu and Jiaolong Kong and Yi Li},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00029}
}