一句话：首次大规模实证研究LLM软件工程智能体的“思考-行动-结果”轨迹，揭示成功与失败的行为模式与反模式。
基于LLM的智能体被日益用于自动化程序修复与问题解决，但其内部决策过程尚缺乏系统研究，限制了对失败模式的理解。本文对三种先进智能体（RepairAgent、AutoCodeRover、OpenHands）的交互日志进行大规模实证研究，将其统一为通用格式，共收集120条轨迹与2822次聚焦于程序修复和问题解决的LLM交互。研究方法结合结构性质、动作模式与token用量的定量分析，以及推理连贯性与反馈整合的定性评估。研究识别出迭代次数、token消耗、重复动作序列、思考-行动-结果的语义连贯性等关键轨迹特征，并发现可区分成功与失败执行的行为基元与反模式，为提示策略、故障诊断与反模式检测提供可操作见解。作者公开数据集与标注框架，以支持透明且鲁棒的自主软件工程智能体的后续研究。
Large Language Model (LLM)-based agents are increasingly employed to automate complex software engineering tasks, such as program repair and issue resolution. These agents operate by autonomously generating natural language thoughts, invoking external tools, and iteratively refining their solutions. Despite their widespread adoption, the internal decision-making processes of these agents remain largely unexplored, limiting our understanding of their operational dynamics and failure modes. In this paper, we present a large-scale empirical study of the thought-action-result trajectories of three state-of-the-art LLM-based agents: RepairAgent, AutoCodeRover, and OpenHands. We unify their interaction logs into a common format, capturing 120 trajectories and 2,822 LLM interactions focused on program repair and issue resolution. Our study combines quantitative analyses of structural properties, action patterns, and token usage with qualitative assessments of reasoning coherence and feedback integration. We identify key trajectory characteristics, such as iteration counts and token consumption, recurring action sequences, and the semantic coherence of thoughts, actions, and their results. Our findings reveal behavioral motifs and anti-patterns that distinguish successful from failed executions, providing actionable insights for improving agent design, including prompting strategies, failure diagnosis, and anti-pattern detection. We release our dataset and annotation framework to support further research on transparent and robust autonomous software engineering agents.
DOI 原文 ·
@inproceedings{paperbot9413,
title = {Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories},
author = {Islem Bouzenia and Michael Pradel},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00234}
}