一句话：提出 Appaction，用多模态学习理解移动 GUI 控件并自动生成有效输入，实现大规模表单测试。
工业界移动应用 GUI 测试仍依赖大量人工，尤其商业应用中的表单界面多样复杂，理解 GUI 并编写测试脚本成本很高。Appaction 是美团开发的基于学习的自动 GUI 交互方法，采用多模态模型学习人类感知 GUI 的经验，整体分析目标界面并理解每个输入的语义，进而推断能够正确交互的有效输入事件，自动驱动目标应用执行。论文汇报了在美团多个热门商业应用上的落地经验，结果表明 Appaction 在 GUI 分析上有效，并能对大量表单页面执行正确交互。
In industrial practice, GUI (Graphic User Interface) testing of mobile apps still inevitably relies on huge manual efforts. The major efforts are those on understanding the GUIs, so that testing scripts can be written accordingly. Quality assurance could therefore be very labor-intensive, especially for modern commercial mobile apps, where one may include tremendous, diverse, and complex GUIs, e.g., those for placing orders of different commercial items. To reduce such human efforts, we propose Appaction, a learning-based automatic GUI interaction approach we developed for Meituan, one of the largest E-commerce providers with over 600 million users. Appaction can automatically analyze the target GUI and understand what each input of the GUI is about, so that corresponding valid inputs can be entered accordingly. To this end, Appaction adopts a multi-modal model to learn from human experiences in perceiving a GUI. This allows it to infer corresponding valid input events that can properly interact with the GUI. In this way, the target app can be effectively exercised. We present our experiences in Meituan on applying Appaction to popular commercial apps. We demonstrate the effectiveness of Appaction in GUI analysis, and it can perform correct interactions for numerous form pages.
@inproceedings{HuGHZTGCZ23,
title = {Appaction: Automatic GUI Interaction for Mobile Apps via Holistic Widget Perception},
author = {Yongxiang Hu and Jiazhen Gu and Shuqing Hu and Yu Zhang and Wenjie Tian and Shiyu Guo and Chaoyi Chen and Yangfan Zhou},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3613885}
}