一句话：提出 SpecOps，用四个 LLM 专家智能体全自动测试真实 GUI 环境中的 AI 智能体。
大语言模型驱动的自主智能体正被部署到真实应用中，其行为可靠性至关重要，但现有评估框架依赖人工、局限于模拟环境，且缺乏对复杂多模态真实智能体的测试。SpecOps 将测试流程分解为测试用例生成、环境搭建、测试执行和验证四个阶段，每个阶段由专门的 LLM 智能体负责，从而解决端到端任务连贯性、鲁棒错误处理以及跨 CLI 工具、Web 应用和浏览器扩展的适应性问题。在五个真实智能体上的综合评估表明，SpecOps 在规划准确性、执行成功率和缺陷检测效果上均优于 AutoGPT 等通用智能体系统和 LLM 编写的自动化脚本，共发现 164 个真实缺陷，F1 分数为 0.89；每次测试成本低于 0.73 美元、运行时间少于八分钟，展示了实际可行性。
Autonomous AI agents powered by large language models (LLMs) are increasingly deployed in real-world applications, where reliable and robust behavior is critical. However, existing agent evaluation frameworks either rely heavily on manual efforts, operate within simulated environments, or lack focus on testing complex, multimodal, real-world agents. We introduce SpecOps, a novel, fully automated testing framework designed to evaluate GUI-based AI agents in real-world environments. SpecOps decomposes the testing process into four specialized phases—test case generation, environment setup, test execution, and validation—each handled by a distinct LLM-based specialist agent. This structured architecture addresses key challenges including end-to-end task coherence, robust error handling, and adaptability across diverse agent platforms including CLI tools, web apps, and browser extensions. In comprehensive evaluations across five diverse real-world agents, SpecOps outperforms baselines including general-purpose agentic systems such as AutoGPT and LLM-crafted automation scripts in planning accuracy, execution success, and bug detection effectiveness. SpecOps identifies 164 true bugs in the real-world agents with an F1 score of 0.89. With a cost of under $0.73 and a runtime of under eight minutes per test, it demonstrates its practical viability and superiority in automated, real-world agent testing.
@inproceedings{abs-2603-10268,
title = {SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environments},
author = {Syed Yusuf Ahmed and Shiwei Feng and Chanwoo Bae and Calix Barrus and Xiangyu Zhang},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3787778}
}