一句话：提出 Guardian 运行时框架，通过约束 LLM 的 UI 动作空间与失效重规划，提升基于 LLM 的自动 UI 测试有效性。
基于功能的移动 UI 测试对应用质量至关重要，但人工创建测试成本高。自动化方法仅给高层测试目标，就需探索被测应用并找到达成目标的 UI 事件序列；该任务类似 AI 规划，但作者发现 LLM 难以遵循 UI 测试的特定指令，也难以根据新信息重规划。为此，论文提出 Guardian 运行时框架，从 LLM 卸载计算：细化 LLM 可规划的 UI 动作空间以强制指令遵循；并检查新信息是否使先前规划失效，若失效则移除相应动作、恢复应用状态并提示 LLM 重新规划。作者用 ChatGPT 实例化 Guardian，并在含 23 个流行应用、58 个任务的 FestiVal 基准上评估，结果显示其达到 48.3（摘要截断）。
Tests for feature-based UI testing have been indispensable for ensuring the quality of mobile applications (apps for short). The high manual labor costs to create such tests have led to a strong interest in automated feature-based UI testing, where an approach automatically explores the App under Test (AUT) to find correct sequences of UI events achieving the target test objective, given only a high-level test objective description. Given that the task of automated feature-based UI testing resembles conventional AI planning problems, large language models (LLMs), known for their effectiveness in AI planning, could be ideal for this task. However, our study reveals that LLMs struggle with following specific instructions for UI testing and replanning based on new information. This limitation results in reduced effectiveness of LLM-driven solutions for automated feature-based UI testing, despite the use of advanced prompting techniques. Toward addressing the preceding limitation, we propose Guardian, a runtime system framework to improve the effectiveness of automated feature-based UI testing by offloading computational tasks from LLMs with two major strategies. First, Guardian refines UI action space that the LLM can plan over, enforcing the instruction following of the LLM by construction. Second, Guardian deliberately checks whether the gradually enriched information invalidates previous planning by the LLM. Guardian removes the invalidated UI actions from the UI action space that the LLM can plan over, restores the state of the AUT to the state before the execution of the invalidated UI actions, and prompts the LLM to re-plan with the new UI action space. We instantiate Guardian with ChatGPT and construct a benchmark named FestiVal with 58 tasks from 23 highly popular apps. Evaluation results on FestiVal show that Guardian achieves 48.3
DOI 原文 ·
@inproceedings{paperbot8526,
title = {Guardian: A Runtime Framework for LLM-Based UI Exploration},
author = {Dezhi Ran and Hao Wang and Zihe Song and Mengzhou Wu and Yuan Cao and Ying Zhang and Wei Yang and Tao Xie},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680334}
}