一句话：提出 CI-Bench 统一基准框架，用真实 CI/CD 构建失败数据动态评测 LLM 程序修复工具。
大语言模型在复杂软件工程任务上展现出潜力，但其严格评估需要来自真实工程流程的大规模、持续更新的数据。现有的 CI/CD 数据集（如 BugSwarm）蕴含不断演化的构建失败完整上下文，却缺少统一的评测框架。本文提出 CI-Bench：它从 BugSwarm 检索数据、解析构建日志并构造合适的提示词，再调用基于 LLM 的程序修复工具；同时提供执行器，在与原始构建过程完全相同的环境中进行动态评估。作者用 GPT-4o、Claude-3.5-Sonnet 与 Deepseek-V3 作为基础模型，在 100 个真实 CI/CD 失败上评测了 Agentless、SWE-Agent 与 AutoCodeRover 三个先进修复工具，其生成正确补丁的成功率分别最高达 28%、35% 与 13%。CI-Bench 已在 GitHub 开源并提供工具演示。
Large language models (LLMs) have demonstrated their potential in performing complex software engineering (SE) tasks. Rigorous evaluation of LLMs and LLM-based tools requires massive, up-to-date data derived from real-world SE processes. Existing continuous integration & delivery (CI/CD) datasets, such as BugSwarm, provide evolving data mined from software build processes, including complete context of build failures. To harness the CI/CD data, we propose CI-Bench, a unified benchmarking framework designed to evaluate LLM-based program repair tools on software failures from CI/CD processes. CI-Bench retrieves data from the BugSwarm dataset, parses the build logs, and constructs appropriate prompts before invoking LLM-based program repair tools. Additionally, CI-Bench includes an executor to facilitate dynamic evaluation within an environment identical to the original build process. With CI-Bench, we evaluate three state-of-the-art LLM-based program repair tools, Agentless, SWE-Agent, and AutoCodeRover, on a code repair task involving 100 real-world CI/CD failures using GPT-4o, Claude-3.5-Sonnet, and Deepseek-V3 as foundation models. Our evaluation shows that Agentless, SWE-Agent, and AutoCodeRover achieve success rates of up to 28%, 35%, and 13%, respectively, in generating correct patches. CI-Bench is available on GitHub at https://github.com/bugswarm/ci-bench, with an accompanying tool demo at https://youtu.be/BM0K-P38MOg.