一句话：将移动 GUI 测试建模为问答任务，用 LLM 结合功能感知记忆提示生成并迭代测试脚本，显著提升覆盖率与缺陷检测。
移动应用自动化 GUI 测试对保障应用质量至关重要，但现有基于学习的方法覆盖率低、泛化不足且严重依赖训练数据。本文受 ChatGPT 等大语言模型在自然语言理解与问答上的成功启发，将移动 GUI 测试建模为问答任务，提出 GPTDroid：把 GUI 页面信息传给 LLM 生成测试脚本，执行后把应用反馈继续返回给 LLM，迭代该过程。框架还引入功能感知的记忆提示机制，使 LLM 保留整个测试过程的知识，并基于功能做长期推理以指导探索。在 Google Play 的 93 个应用上评估，其活动覆盖率比最佳基线高 32%，发现缺陷多 31% 且速度更快；并在 Google Play 上发现 53 个新缺陷，其中 35 个已被确认修复。
Automated Graphical User Interface (GUI) testing plays a crucial role in ensuring app quality, especially as mobile applications have become an integral part of our daily lives. Despite the growing popularity of learning-based techniques in automated GUI testing due to their ability to generate human-like interactions, they still suffer from several limitations, such as low testing coverage, inadequate generalization capabilities, and heavy reliance on training data. Inspired by the success of Large Language Models (LLMs) like ChatGPT in natural language understanding and question answering, we formulate the mobile GUI testing problem as a Q&A task. We propose GPTDroid, asking LLM to chat with the mobile apps by passing the GUI page information to LLM to elicit testing scripts, and executing them to keep passing the app feedback to LLM, iterating the whole process. Within this framework, we have also introduced a functionality-aware memory prompting mechanism that equips the LLM with the ability to retain testing knowledge of the whole process and conduct long-term, functionality-based reasoning to guide exploration. We evaluate it on 93 apps from Google Play and demonstrate that it outperforms the best baseline by 32% in activity coverage, and detects 31% more bugs at a faster rate. Moreover, GPTDroid identifies 53 new bugs on Google Play, of which 35 have been confirmed and fixed.
@inproceedings{paperbot11304,
title = {Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions},
author = {Zhe Liu and Chunyang Chen and Junjie Wang and Mengzhuo Chen and Boyu Wu and Xing Che and Dandan Wang and Qing Wang},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639180}
}