paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT

通过对话的自动程序修复:使用 ChatGPT 以每个 0.42 美元修复 337 个缺陷中的 162 个
ISSTA Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis2024匹配度 4/10 · 论文聚焦 LLM 对话式自动程序修复(APR),属于 AI4SE/程序修复,与用户核心关注的静态分析、形式化验证、浮点程序分析、贝叶斯程序分析均不直接匹配;仅与“LLM/Agent 作为工具改进程序相关任务”有宽泛交集,且主要基于 ChatGPT 验证,未体现当前较强模型。引用 136 inbox
Chunqiu Steven Xia, Lingming Zhang
crossref: 2026-09-12

AI 速览

一句话:提出 ChatRepair,首个全自动对话驱动 APR 方法,利用 ChatGPT 结合测试反馈迭代生成补丁,显著提升修复效果并降低成本。

传统自动程序修复依赖手工或挖掘的修复模式,补丁多样性不足且难以泛化;现有基于大语言模型的 APR 工具虽效果领先,但仍遵循先生成大量补丁再验证的范式,导致重复错误补丁,并忽略测试失败与合理补丁中的关键信息。本文提出 ChatRepair,首个全自动对话驱动的 APR 方法,将补丁生成与即时反馈交错进行。它首先向 LLM 提供相关测试失败信息;对于失败补丁,结合其对应测试失败构造新提示以避免重复错误;对于通过测试的合理补丁,则要求 LLM 生成变体,从成功中学习以增加获得正确补丁的机会。基于 ChatGPT 实现后,在 Defects4j 1.2 和 2.0 上分别取得 114 和 48 个正确修复,达到新 SOTA;按 ChatGPT 访问成本计算,能以每个 0.42 美元修复 337 个缺陷中的 162 个。

问题
传统 APR 依赖手工或挖掘的修复模式,补丁多样性差且难以泛化;现有 LLM-based APR 仍采用 Generate and Validate 范式,生成大量重复错误补丁,并忽略测试失败与合理补丁中的信息。
方法
提出 ChatRepair,首个全自动对话驱动 APR 方法:先用相关测试失败信息启动 LLM;对失败补丁结合测试失败构造新提示以避免重蹈覆辙;对合理补丁要求生成变体,从成功中学习并迭代生成更多补丁。
评估
使用 ChatGPT 实现 ChatRepair,在广泛研究的 Defects4j 数据集上评估,在 Defects4j 1.2 和 2.0 上分别获得 114 和 48 个正确修复,达到新的 SOTA;按 ChatGPT 成本计算,337 个缺陷中修复 162 个,每个约 0.42 美元。
原文摘要(Abstract)

Automated Program Repair (APR) aims to automatically generate patches for buggy programs. Traditional APR techniques suffer from a lack of patch variety as they rely heavily on handcrafted or mined bug fixing patterns and cannot easily generalize to other bug/fix types. To address this limitation, recent APR work has been focused on leveraging modern Large Language Models (LLMs) to directly generate patches for APR. Such LLM-based APR tools work by first constructing an input prompt built using the original buggy code and then querying the LLM to either fill-in (cloze-style APR) the correct code at the bug location or to produce a completely new code snippet as the patch. While the LLM-based APR tools are able to achieve state-of-the-art results, they still follow the classic Generate and Validate (GV) repair paradigm of first generating lots of patches by sampling from the same initial prompt and then validating each one afterwards. This not only leads to many repeated patches that are incorrect, but also misses the crucial and yet previously ignored information in test failures as well as in plausible patches. To address these aforementioned limitations, we propose ChatRepair, the first fully automated conversation-driven APR approach that interleaves patch generation with instant feedback to perform APR in a conversational style. ChatRepair first feeds the LLM with relevant test failure information to start with, and then learns from both failures and successes of earlier patching attempts of the same bug for more powerful APR. For earlier patches that failed to pass all tests, we combine the incorrect patches with their corresponding relevant test failure information to construct a new prompt for the LLM to generate the next patch. In this way, we can avoid making the same mistakes. For earlier patches that passed all the tests (i.e., plausible patches), we further ask the LLM to generate alternative variations of the original plausible patches. In this way, we can further build on and learn from earlier successes to generate more plausible patches to increase the chance of having correct patches. While our approach is general, we implement ChatRepair using state-of-the-art dialogue-based LLM – ChatGPT. Our evaluation on the widely studied Defects4j dataset shows that ChatRepair is able to achieve the new state-of-the-art in repair performance, achieving 114 and 48 correct fixes on Defects4j 1.2 and 2.0 respectively. By calculating the cost of accessing ChatGPT, we can fix 162 out of 337 bugs for $0.42 each!

链接与引用

DOI 原文 · PDF(开放获取) ·

BibTeX
@inproceedings{paperbot8523,
  title = {Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT},
  author = {Chunqiu Steven Xia and Lingming Zhang},
  booktitle = {ISSTA},
  year = {2024},
  doi = {10.1145/3650212.3680323}
}