一句话：定义并系统研究 LLM 程序修复中的事实选择问题，提出按缺陷选择提示事实的统计模型 Maniple。
LLM 程序修复常通过在提示中加入栈轨迹、GitHub issue 等 bug 相关事实来提升效果，但哪些事实、多少事实应被包含仍不清楚。本文在 BugsInPy 的 314 个 Python bug 上，用超过 19K 个提示系统评测七类事实的组合，发现每类事实都有帮助，但提示修复效果随事实数量呈非单调变化，且不存在通用最优事实集。作者据此定义事实选择问题，即针对具体任务实例挑选最优事实集合，并提出统计模型 Maniple。在 157 个测试 bug 上，Maniple 修复 88 个，较最佳通用事实配置提升 17%。
Recent research has shown that incorporating bug-related facts, such as stack traces and GitHub issues, into prompts enhances the bug-fixing capabilities of large language models (LLMs). Considering the ever-increasing context window of these models, a critical question arises: what and how many facts should be included in prompts to maximise the chance of correctly fixing bugs? To answer this question, we conducted a large-scale study, employing over 19K prompts featuring various combinations of seven diverse facts to rectify 314 bugs from open -source Python projects within the BugsInPy benchmark. Our findings revealed that each fact, ranging from simple syntactic details like code context to semantic information previously unexplored in the context of LLMs such as angelic values, is beneficial. Specifically, each fact aids in fixing some bugs that would remain unresolved or only be fixed with a low success rate without it. Importantly, we discovered that the effectiveness of program repair prompts is non-monotonic over the number of used facts; using too many facts leads to subpar outcomes. These insights led us to define the fact selection problem: determining the optimal set of facts for inclusion in a prompt to maximise LLM's performance on a given task instance. We found that there is no one-size-fits-all set of facts for bug repair. Therefore, we developed a basic statistical model, named Maniple, which selects facts specific to a given bug to include in the prompt. This model significantly surpasses the performance of the best generic fact set. To underscore the significance of the fact selection problem, we benchmarked Maniple against the state-of-the-art zero-shot, non-conversational LLM-based bug repair methods. On our testing dataset of 157 bugs, Maniple repairs 88 bugs, 17% above the best configuration.
DOI 原文 ·
@inproceedings{paperbot8704,
title = {The Fact Selection Problem in LLM-Based Program Repair},
author = {Nikhil Parasaram and Huijie Yan and Boyu Yang and Zineb Flahy and Abriele Qudsi and Damian Ziaber and Earl T. Barr and Sergey Mechtaev},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00162}
}