一句话：提出基于多模态 LLM 智能体的框架，自动验证 GUI 原型对自然语言需求的满足情况并生成可操作反馈。
GUI 原型是交互系统早期需求获取的关键载体，但验证其实现是否满足自然语言（NL）需求十分困难：传统与 LLM 驱动的 GUI 测试方法难以应对现代界面的复杂性，且普遍缺乏可操作反馈与对自动化开发智能体的有效集成。本文提出 GUISpector，一个利用多模态（M）LLM 智能体自动验证 GUI 原型中 NL 需求的框架：首先适配 MLLM 智能体以解释并操作化 NL 需求，自主规划并执行跨 GUI 应用的验证轨迹；其次系统化地从验证过程中抽取细粒度 NL 反馈，为开发者提供可操作的洞察，用于迭代精化 GUI 制品或直接驱动 LLM 代码生成，形成闭环；最后给出统一上述能力的集成工具界面，支持监督验证运行、检查智能体推理并管理端到端需求验证流程。作者在涵盖多种 GUI 应用的 150 条需求与 900 条验收标准上评估，结果显示能有效检测需求的满足与违反，并展现将可操作反馈无缝融入 LLM 驱动自动化开发工作流的潜力。
Graphical user interfaces (GUIs) are foundational to interactive systems and play a pivotal role in early requirements elicitation through prototyping. Ensuring that GUI implementations fulfill natural language (NL) requirements is essential for robust software engineering, especially as LLM-driven programming agents become increasingly integrated into development workflows. Existing GUI testing approaches, whether traditional or LLM-driven, often fall short in handling the complexity of modern interfaces, and typically lack actionable feedback and effective integration with automated development agents. In this paper, we introduce GUISpector, a novel framework that leverages a multi-modal (M)LLM-based agent for the automated verification of NL requirements in GUI prototypes. First, GUISpector adapts a MLLM agent to interpret and operationalize NL requirements, enabling to autonomously plan and execute verification trajectories across GUI applications. Second, GUISpector systematically extracts detailed NL feedback from the agent’s verification process, providing developers with actionable insights that can be used to iteratively refine the GUI artifact or directly inform LLM-based code generation in a closed feedback loop. Third, we present an integrated tool that unifies these capabilities, offering practitioners an accessible interface for supervising verification runs, inspecting agent rationales and managing the end-to-end requirements verification process. We evaluated GUISpector on a comprehensive set of 150 requirements based on 900 acceptance criteria annotations across diverse GUI applications, demonstrating effective detection of requirement satisfaction and violations and highlighting its potential for seamless integration of actionable feedback into automated LLM-driven development workflows. The video presentation of GUISpector is available at: https://youtu.be/JByYF6BNQeE, showcasing its main capabilities.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2510-04791,
title = {GUISpector: An MLLM Agent Framework for Automated Verification of Natural Language Requirements in GUI Prototypes},
author = {Kristian Kolthoff and Felix Kretzer and Simone Paolo Ponzetto and Alexander Maedche and Christian Bartelt},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3774748.3787611}
}