一句话：提出认知可观测性概念与 Watson 框架，用提示归因回溯推断 LLM 智能体的推理轨迹，以支持调试与纠正。
LLM 被集成到自主系统中形成一类新软件 Agentware，其智能体在软件工程、客服、数据分析等领域执行开放式任务。但高度自主与不透明的推理过程令传统软件可观测性方法失效。论文提出认知可观测性，即恢复并检查智能体决策背后隐式推理的能力，并给出通用框架 Watson。Watson 不改变智能体行为，利用提示归因（prompt attribution）技术回溯推断推理轨迹。作者在 MMLU 基准以及 SWE-bench-lite 上的 AutoCodeRover、OpenHands 智能体上，从手动调试与自动纠正两种场景进行评估。结果表明，Watson 在静态与动态设置下均能揭示可操作的推理洞见并支持针对性干预，提升了 Agentware 系统的透明性与可靠性。
Large language models (LLMs) are increasingly integrated into autonomous systems, giving rise to a new class of software known as Agentware, where LLM-powered agents perform complex, open-ended tasks in domains such as software engineering, customer service, and data analysis. However, their high autonomy and opaque reasoning processes pose significant challenges for traditional software observability methods. To address this, we introduce the concept of cognitive observability—the ability to recover and inspect the implicit reasoning behind agent decisions. We present Watson, a general-purpose framework for observing the reasoning processes of fast-thinking LLM agents without altering their behavior. Watson retroactively infers reasoning traces using prompt attribution techniques. We evaluate Watson in both manual debugging and automated correction scenarios across the MMLU benchmark and the AutoCodeRover and OpenHands agents on the SWE-bench-lite dataset. In both static and dynamic settings, Watson surfaces actionable reasoning insights and supports targeted interventions, demonstrating its practical utility for improving transparency and reliability in Agentware systems.
DOI 原文 ·
@inproceedings{paperbot9282,
title = {Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents},
author = {Benjamin Rombaut and Sogol Masoumzadeh and Kirill Vasilevski and Dayi Lin and Ahmed E. Hassan},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00067}
}