一句话：提出 AgentOps 框架，用六阶段自动化流水线观察、分析与优化 Agentic AI 系统以驾驭不确定性。
LLM 驱动的 Agentic AI 系统通过记忆、工具和动态规划执行复杂自适应工作流，但概率推理、演化记忆与流动执行路径带来新型不确定性；传统软件可观测性与运维实践不足。本文提出 AgentOps 愿景框架，围绕开发者、测试者、SRE 和业务用户四类角色识别全生命周期需求，并给出六阶段流水线：行为观察、指标收集、问题检测、根因分析、优化建议与运行时自动化。其核心观点是自动化并非消除不确定性，而是驯服不确定性，以支撑安全、自适应且可自我改进的 AI 系统。该工作为愿景性框架，未报告实证评估。
Large Language Models (LLMs) are increasingly deployed within agentic systems—collections of interacting, LLM-powered agents that execute complex, adaptive workflows using memory, tools, and dynamic planning. While enabling powerful new capabilities, these systems also introduce unique forms of uncertainty stemming from probabilistic reasoning, evolving memory states, and fluid execution paths. Traditional software observability and operations practices fall short in addressing these challenges.This paper presents our vision of AgentOps: a comprehensive framework for observing, analyzing, optimizing, and automating operation of agentic AI systems. We identify distinct needs across four key roles—developers, testers, site reliability engineers (SREs), and business users—each of whom engages with the system at different points in its lifecycle. We present the AgentOps Automation Pipeline, a six-stage process encompassing behavior observation, metric collection, issue detection, root cause analysis, optimized recommendations, and runtime automation. Throughout, we emphasize the critical role of automation in managing uncertainty and enabling self-improving AI systems—not by eliminating uncertainty, but by taming it to ensure safe, adaptive, and effective operation.
DOI 原文 ·
@inproceedings{paperbot9341,
title = {Taming Uncertainty via Automation: Observing, Analyzing, and Optimizing Agentic AI Systems},
author = {Dany Moshkovich and Sergey Zeltyn},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00327}
}