一句话：结合 STPA 危害分析与能力增强 MCP，为 LLM Agent 的工具调用提供形式化安全护栏。
基于大语言模型的 AI Agent 通过调用数据源、API、搜索引擎、代码沙箱乃至其他 Agent 等工具来完成复杂任务，但也可能触发非预期的工具交互，导致敏感数据泄露或关键记录被覆盖，这在企业场景中不可接受。现有的基于模型的防护措施虽能提升可靠性，却无法保证系统安全；信息流控制（IFC）与时序约束等方法虽能提供保证，但往往依赖大量人工标注。本文提出一套流程：首先用系统理论过程分析（STPA）识别 Agent 工作流中的危害并导出安全需求，再将其形式化为对数据流与工具调用序列的可执行规范。为支撑这一流程，作者引入能力增强的模型上下文协议（MCP）框架，要求对 capability、confidentiality 与 trust level 进行结构化标注。其目标是把 Agent 安全从临时的可靠性修补转变为具形式保证的主动护栏，减少对用户逐一确认的依赖，使自主性成为有意的设计选择。
Large language model (LLM)-based AI agents extend LLM capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents. While this empowers agents to perform complex tasks, LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records, which are unacceptable in enterprise contexts. Current approaches to mitigate these risks, such as model-based safeguards, enhance agents’ reliability but cannot guarantee system safety. Methods like information flow control (IFC) and temporal constraints aim to provide guarantees but often require extensive human annotation. We propose a process that starts with applying System-Theoretic Process Analysis (STPA) to identify hazards in agent workflows, derive safety requirements, and formalize them as enforceable specifications on data flows and tool sequences. To enable this, we introduce a capability-enhanced Model Context Protocol (MCP) framework that requires structured labels on capabilities, confidentiality, and trust level. Together, these contributions aim to shift LLM-based agent safety from ad hoc reliability fixes to proactive guardrails with formal guarantees, while reducing dependence on user confirmation and making autonomy a deliberate design choice.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{DoshiHXKKK26,
title = {Towards Verifiably Safe Tool Use for LLM Agents},
author = {Aarya Doshi and Yining Hong and Congying Xu and Eunsuk Kang and Alexandros Kapravelos and Christian Kästner},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3786582.3786839}
}