一句话:提出以执行轨迹为中心的过程感知框架,用 Graphectory 和 Langutory 刻画编程智能体的行为模式与计划遵循度。
自主编程智能体通常仅以是否解决软件任务来评估,这无法揭示导致成败的执行过程。本文提出以过程为中心的轨迹评估方法:Graphectory 捕捉智能体执行的结构组织,Langutory 将底层动作抽象为软件工程阶段以刻画问题解决策略,并通过计划遵循度将观察到的执行与预期工作流关联。在数千条轨迹上的分析揭示了仅靠结果无法解释的行为模式,包括成功运行中持续存在的低效,以及智能体将计划转化为执行时的显著差异。这些发现凸显理解智能体执行过程的重要性,为软件智能体的过程感知优化奠定基础。
Autonomous programming agents are commonly evaluated by whether they solve software tasks, providing limited insight into the execution processes that lead to success or failure. We present a process-centric approach to evaluating agent behavior through trajectories. We introduce Graphectory, which captures the structural organization of agent execution, and Langutory, which abstracts low-level actions into software engineering phases to characterize problem-solving strategies. We further relate observed execution to intended workflows through plan compliance. Across thousands of trajectories, our analysis uncovers behavioral patterns that outcomes alone cannot explain, including persistent inefficiencies in successful runs and substantial variation in how agents translate plans into execution. These findings highlight the importance of understanding how agents execute, laying the foundation for the process-aware optimization of software agents.