一句话：构建 34 个可编程任务基准，评测 LLM 自主智能体并给出三层失败原因分类与改进建议
由大语言模型驱动的自主智能体在复杂任务自动化上展现出潜力，但现有评测大多只报告成功率，缺乏对系统内部交互、通信机制与失败原因的系统性分析。为弥补这一空白，本文提出一个包含 34 个代表性可编程任务的基准，用于严格评估自主智能体。基于该基准，作者评测了三个流行的开源智能体框架与两种 LLM 后端，观察到任务完成率约为 50%。通过深入的失败分析，论文给出与任务阶段对齐的三层失败原因分类，指出规划错误、任务执行问题与错误响应生成三类主要成因，并据此提出增强智能体规划与自我诊断能力的可操作改进建议。该失败分类与缓解建议为未来构建更鲁棒、更有效的自主智能体系统提供了实证基础。
Autonomous agent systems powered by Large Language Models (LLMs) have demonstrated promising capabilities in automating complex tasks. However, current evaluations largely rely on success rates without systematically analyzing the interactions, communication mechanisms, and failure causes within these systems. To bridge this gap, we present a benchmark of 34 representative programmable tasks designed to rigorously assess autonomous agents. Using this benchmark, we evaluate three popular open-source agent frameworks combined with two LLM backbones, observing a task completion rate of approximately 50%. Through in-depth failure analysis, we develop a three-tier taxonomy of failure causes aligned with task phases, highlighting planning errors, task execution issues, and incorrect response generation. Based on these insights, we propose actionable improvements to enhance agent planning and self-diagnosis capabilities. Our failure taxonomy, together with mitigation advice, provides an empirical foundation for developing more robust and effective autonomous agent systems in the future.
DOI 原文 ·
@inproceedings{paperbot9255,
title = {Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks},
author = {Ruofan Lu and Yichen Li and Yintong Huo},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00330}
}