一句话：实证表明LLM修复循环前3-4轮即贡献绝大部分收益，修复预算应作为显式实验变量。
迭代式修复循环已成为基于LLM的软件工程系统的核心设计模式：工作流反复生成、验证并借助编译错误或测试失败等反馈修复产物。然而，修复循环的迭代次数上限对效果的影响仍缺乏理解，既有工作大多采用固定甚至任意的修复预算。本文在代码生成、测试生成与代码翻译等多类软件工程任务上，选取若干代表性工作流、数据集与当代低成本LLM，系统考察修复循环的有效性。实验观察到一致的收益递减模式：前3至4轮修复即带来大部分可获得的性能提升，后续迭代仅贡献边际改进。此外，在所评估的模型中，修复行为受工作流编排与反馈设计的影响，比受底层模型本身的影响更强。作者据此指出，修复预算会直接影响评估结果、计算成本、运行时间与可复现性，应被视为显式的实验变量。
Iterative repair loops have become a core design pattern in LLM-based software engineering systems. These workflows repeatedly generate, validate, and repair artifacts using feedback such as compiler errors or test failures. Despite their widespread use, the impact of repair-loop iteration limits remains poorly understood, as most prior work adopts fixed, often arbitrary, repair budgets. We study repair-loop effectiveness across multiple software engineering tasks, including code generation, test generation, and code translation. Across several representative workflows, datasets, and contemporary low-cost LLMs, we observe a consistent pattern of diminishing returns: the first three to four repair iterations account for most achievable gains, while later iterations contribute only marginal improvements. Across the evaluated models, repair behavior is influenced more strongly by workflow orchestration and feedback design than by the underlying model itself. These results suggest that repair budgets should be treated as an explicit experimental variable, as they directly affect evaluation outcomes, computational cost, runtime, and reproducibility in LLM-based software engineering research.
@inproceedings{abs-2607-05197,
title = {Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops},
author = {Tobias Kiecker and Eik Reichmann and Hosung Kang and Gabin An and Lars Grunske},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3834547}
}