一句话：首个针对 LLM 智能体框架缺陷的大规模实证研究，给出九类根因与六类症状的分类体系。
LLM 智能体工作流编排框架（如 LangChain）是构建高级 LLM 应用的关键基础设施，但其复杂度导致缺陷不可避免，并会传播至下游应用造成严重失效。本文首先抽象出主流框架的四个关键架构组件：数据预处理、核心 schema、agent 构建与特性模块；继而从 LangChain、LlamaIndex、Haystack 三个流行框架的 1,577 个真实 bug 相关 PR 中提取 1,026 个缺陷实例，开展首个该类框架的缺陷实证研究，逐一分析根因、症状与结构组件，形成九类根因与六类症状的系统分类。最后统计缺陷在框架间与组件间的分布及根因—症状关系，揭示出有别于传统软件的独特挑战模式，并为质量保障实践提供可操作建议。
Large Language Models (LLMs) have rapidly gained popularity, transforming research and industry. To support their adoption, LLM agent workflow orchestration frameworks (hereinafter referred to as LLM agent frameworks) like LangChain have become essential for building advanced applications. However, their complexity makes bugs inevitable, and these bugs can propagate to downstream applications, causing severe failures or unintended behaviors. In this paper, we first present an abstraction of the structure of mainstream LLM agent frameworks, identifying four key architectural components: data preprocessing, core schema, agent construction, and featured modules. Building on this abstraction, we conduct the first empirical study on LLM agent framework bugs, analyzing 1,026 bug instances extracted from 1,577 real-world bug-related GitHub pull requests (PRs) from three popular LLM agent frameworks: LangChain, LlamaIndex, and Haystack. For each bug, we examine its root cause, symptom, and structural component, providing a systematic taxonomy of nine root causes and six symptom categories. Finally, leveraging the framework structure abstraction and the large-scale empirical study, we perform detailed statistical analysis in terms of the distribution of bugs in different frameworks, the distribution across different framework components, and the relationship between root cause and symptom. The analysis reveals unique challenge patterns compared to traditional software, providing actionable guidance for practitioners on quality assurance.
DOI 原文 ·
@inproceedings{paperbot9481,
title = {A Characterization Study of Bugs in LLM Agent Workflow Orchestration Frameworks},
author = {Ziluo Xue and Yanjie Zhao and Shenao Wang and Kai Chen and Haoyu Wang},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00278}
}