一句话：对 SWE-Bench 上 8 个智能体的求解轨迹与测试日志做过程导向错误分析，揭示执行错误与修复成功率的关系。
现有对基于大语言模型的软件开发智能体的评测多聚焦于最终代码的静态分析，难以刻画其多步推理、工具调用与迭代调试的动态过程。本文对 SWE-Bench 基准中 500 个真实 GitHub issue 上 8 个顶尖智能体的 3,977 条求解阶段轨迹与 3,931 条测试阶段日志开展实证研究。探索性分析表明，问题解决阶段的 Python 执行错误与更低的解决率和更高的推理开销相关；作者识别出最普遍的错误（如 ModuleNotFoundError、TypeError），并指出 OSError 与数据库相关错误（如 IntegrityError）需要显著更多的调试投入。此外，研究还发现 3 个影响基准公平性与准确性的 SWE-Bench 平台缺陷，已报告并被维护者确认；数据集与分析脚本已公开。
AI-driven software development has rapidly advanced with the emergence of software development agents that leverage large language models (LLMs) to tackle complex, repository-level software engineering tasks. These agents go beyond just generation of final code; they engage in multi-step reasoning, utilize various tools for code modification and debugging, and interact with execution environments to diagnose and iteratively resolve issues. However, most existing evaluations focus primarily on static analyses of final code outputs, yielding limited insights into the agents’ dynamic problem-solving processes. To fill this gap, we conduct an in-depth empirical study on 3,977 solving-phase trajectories and 3,931 testing-phase logs from 8 top-ranked agents evaluated on 500 GitHub issues in the SWE-Bench benchmark. Our exploratory analysis shows that Python execution errors during the issue resolution phase correlate with lower resolution rates and increased reasoning overheads. We have identified the most prevalent errors—such as ModuleNotFoundError and TypeError—and highlighted particularly challenging errors like OSError and database-related issues (e.g., IntegrityError) that demand significantly more debugging effort. Furthermore, we have discovered 3 bugs in the SWE-Bench platform that affect benchmark fairness and accuracy; these issues have been reported to and confirmed by the maintainers. To promote transparency and foster future research, we publicly share our datasets and analysis scripts.
@inproceedings{paperbot11602,
title = {Beyond Final Code: A Process-Oriented Error Analysis of Software Development Agents in Real-World GitHub Scenarios},
author = {Zhi Chen and Wei Ma and Lingxiao Jiang},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3773140}
}