一句话：结合检索增强LLM与机器学习，为工业级App生成低成本UI自动化测试，微信上以0.34美元实现90%自动化率。
UI自动化测试对移动应用质量至关重要，但基于机器学习的测试生成常出现UI元素匹配错误；LLM的语义理解虽可缓解，却在工业级应用中受限于成本与领域知识。本文提出CAT，将机器学习与LLM及最佳实践结合：依据任务描述，用检索增强生成（RAG）检索工业App使用示例作为few-shot上下文，引导LLM生成具体动作序列；再用机器学习方法将目标元素映射到UI屏幕，LLM作为补充优化器。在微信测试数据集上，CAT以0.34美元成本实现90%的UI自动化，优于现有最佳方法；并已集成到微信真实测试平台，检测出141个bug，改善了开发者的测试流程。
UI automation tests play a crucial role in ensuring the quality of mobile applications. Despite the growing popularity of machine learning techniques to generate these tests, they still face several challenges, such as the mismatch of UI elements. The recent advances in Large Language Models (LLMs) have addressed these issues by leveraging their semantic understanding capabilities. However, a significant gap remains in applying these models to industrial-level app testing, particularly in terms of cost optimization and knowledge limitation. To address this, we introduce CAT to create cost-effective UI automation tests for industry apps by combining machine learning and LLMs with best practices. Given the task description, CAT employs Retrieval Augmented Generation (RAG) to source examples of industrial app usage as the few-shot learning context, assisting LLMs in generating the specific sequence of actions. CAT then employs machine learning techniques, with LLMs serving as a complementary optimizer, to map the target element on the UI screen. Our evaluations on the WeChat testing dataset demonstrate the CAT’s performance and cost-effectiveness, achieving 90% UI automation with $0.34 cost, outperforming the state-of-the-art. We have also integrated our approach into the real-world WeChat testing platform, demonstrating its usefulness in detecting 141 bugs and enhancing the developers’ testing process.CCS CONCEPTS• Software and its engineering → Software testing and debugging.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{FengLJXHLLDA24,
title = {Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChat},
author = {Sidong Feng and Haochuan Lu and Jianqin Jiang and Ting Xiong and Likun Huang and Yinglin Liang and Xiaoqin Li and Yuetang Deng and Aldeida Aleti},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695260}
}