paperbot · PL 论文追踪

全部论文 待读 收藏

Introducing Safe LLM-Agent Execution via Typed Tool Mediation and World-Effect Evaluation

通过类型化工具中介与世界效应评估实现安全的 LLM 智能体执行
ASE Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering2026引用 0 inbox
Jun Zhou, Tao Jia
crossref: 2026-10-09

AI 速览

一句话:提出 AgentOS,用类型化工具中介 VTRM 与宿主侧世界效应评判器,在保持良性任务效用的同时阻断 LLM 智能体的有害世界效应。

LLM 智能体日益作为可执行动作的系统,对文件、数据库、网络和外部服务进行操作,其安全失败不再只是不安全文本,而可能造成未授权的世界效应,如数据删除、受保护资源篡改或秘密外泄。本文提出 AgentOS,一个基于 Buildroot 的 LLM 智能体安全执行环境,其核心组件 Verified Toolchain Reference Monitor(VTRM)在执行前中介模型生成的工具请求,将不可信模型输出转换为类型化请求,并对文件系统、数据库和网络操作实施能力感知与标签感知的决策。AgentOS 还包含宿主侧 World Effect Judge(WEJ),依据文件系统快照、数据库差异、审计日志与网络观测而非模型自报来评估任务完成与安全性。作者在两个 LLM 上使用良性任务与对抗任务套件进行评估:在良性任务效用上 AgentOS 与仅提示词及朴素基线相当;在对抗任务上 AgentOS 阻断全部已评估的有害案例,而两个基线遗留大量未阻断的不安全效应。作者还用 TLA+ 建模 VTRM 核心决策路径并检查代表性的机密性与 fail-closed 性质。

问题
LLM 智能体执行文件、数据库与网络操作时,不安全输出可能转化为数据删除、资源篡改或秘密外泄等未授权的世界效应,而现有基线缺乏对工具请求的有效中介与客观的安全评估。
方法
提出基于 Buildroot 的 AgentOS 执行环境,其 VTRM 将不可信模型输出转换为类型化工具请求,并在执行前对文件系统、数据库和网络操作实施能力感知与标签感知的中介决策;同时设计宿主侧 World Effect Judge,用系统级观测而非模型自报来评估任务完成与安全性,并用 TLA+ 对核心决策路径建模检查机密性与 fail-closed 性质。
评估
在两个 LLM 上使用良性任务与对抗任务套件进行对比评估,以文件系统快照、数据库差异、审计日志与网络观测衡量任务完成和安全性;结果显示良性任务效用与仅提示词及朴素基线相当,而对抗任务中 AgentOS 阻断全部已评估的有害案例,基线则遗留大量未阻断的不安全效应。
原文摘要(Abstract)

LLM-based agents are increasingly used as action-taking systems that operate over files, databases, networks, and external services. In this setting, safety failures can become unauthorized world effects, such as data deletion, protected-resource modification, or secret exfiltration, rather than merely unsafe text outputs. We present AgentOS, a Buildroot-based execution environment for safe tool-using LLM agents. Its core component, the Verified Toolchain Reference Monitor (VTRM), mediates model-generated tool requests before execution by converting untrusted model outputs into typed requests and enforcing capability-aware and label-aware decisions over filesystem, database, and network operations. AgentOS further includes a host-side World Effect Judge (WEJ) for evaluation. WEJ measures task completion and safety using filesystem snapshots, database diffs, audit logs, and network observations rather than model self-report. We evaluate AgentOS on benign and adversarial task suites using two LLMs. Across both models, AgentOS matches prompt-only and naive baselines on benign-task utility. On adversarial tasks, AgentOS blocks all evaluated harmful cases, while both baselines leave substantially more unsafe effects unblocked. We also model the core VTRM decision path in TLA+ and check representative confidentiality and fail-closed properties. These results suggest that typed tool mediation and host-side world-effect evaluation can improve LLM-agent execution safety while preserving benign-task utility. The source code and documentation are publicly available under the MIT License at https://github.com/zhoujune/AgentOS, with an archived artifact at https://doi.org/10.5281/zenodo.20046861; the demonstration screencast is available at https://youtu.be/bqJsTRJbIkU.

链接与引用

DOI 原文 · PDF(开放获取) ·

BibTeX
@inproceedings{paperbot12209,
  title = {Introducing Safe LLM-Agent Execution via Typed Tool Mediation and World-Effect Evaluation},
  author = {Jun Zhou and Tao Jia},
  booktitle = {ASE},
  year = {2026},
  doi = {10.1145/3832783.3834637}
}