一句话：提出在共享 HTTP 层运行时非侵入式注入 LLM API 故障的混沌工程框架，并系统评估智能体系统鲁棒性。
智能体系统的每一次响应都依赖 LLM API，而 API 可能返回服务器错误、截断或损坏内容，这些故障会沿下游智能体传播并导致任务失败，因此评估其鲁棒性对可靠部署至关重要。现有故障注入方法多为离线、需要修改源码，且无法精确修改特定响应字段。本文提出 AgentChaos，一个受控、运行时、非侵入式的 LLM API 故障注入混沌工程框架。由于智能体系统都通过同一 HTTP 接口访问 LLM，作者在该共享层注入故障而无需改源码，并在 content 与 tool call 字段上定义 crash、omission、value 三类故障，运行时拦截与修改 API 响应，同时验证故障是否真正触发以过滤未触发任务、避免低估影响。在多个智能体系统、基准与骨干 LLM 上的 65 种故障配置评估显示，所有系统均出现退化，pass@1 最多下降 50 个百分点，且系统排序跨模型一致，说明鲁棒性取决于系统实现而非模型能力；现有故障诊断方法在故障类型上准确率低于 53%、故障步骤上低于 56%，仍有改进空间。
Agent systems rely on LLM APIs for every response, but these APIs can return server errors, truncated responses, or corrupted content that propagates through downstream agents and causes task failure. Evaluating robustness under these faults is crucial for reliable deployment. Existing fault injection methods are offline, require source code modification, or cannot modify specific response fields. A comprehensive evaluation also requires a systematic fault taxonomy because different fault types affect downstream agents differently. We propose AgentChaos, a chaos engineering framework for controlled, runtime, non-intrusive LLM API fault injection. Since all agent systems access LLMs through the same HTTP interface, we inject faults at this shared layer without modifying source code. We define crash, omission, and value faults on content and tool call fields, intercept and modify LLM API responses at runtime, and verify whether each fault is triggered to filter untriggered tasks and avoid underestimating fault impact. Evaluations across agent systems, benchmarks, and backbone LLMs under 65 fault configurations show that all systems degrade under fault injection, with pass@1 dropping by up to 50 percentage points. The ranking is consistent across models, suggesting that robustness depends on system implementation rather than model capability. Existing fault diagnosis methods achieve below 53% accuracy on fault type and below 56% on fault step, leaving room for improvement. We further reveal practical findings for agent system developers.
@inproceedings{paperbot12585,
title = {AgentChaos: Chaos Engineering for Agent Systems via Programmatic Fault Injection},
author = {Gou Tan and Zhensu Sun and Jieke Shi and Ting Zhang and Zilong He and Qingfu Wu and Shuai Liang and Weifeng Sun and Junda He and Pengfei Chen and Chuanfu Zhang and Lwin Khin Shar and David Lo},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837437}
}