一句话：提出基于视觉机械臂的非侵入式 GUI 探索测试方案，以应对跨平台、屏幕差异与兼容性缺陷挑战。
GUI 测试是软件工程的重要问题，但现有框架大多具有侵入性且仅支持特定平台，难以覆盖不同设备上的嵌入式系统或定制操作系统。已有工作用机械臂替代接口调用，并结合计算机视觉识别 GUI 元素，但仍面临三个挑战：假设 GUI 屏幕固定而无法适应不同屏幕条件；采用 XY 平面机械臂系统，难以灵活模拟人工测试操作；只关注崩溃缺陷而忽略兼容性缺陷。为此，本文提出一种更实用的非侵入式 GUI 探索测试方案，基于视觉技术驱动机械臂执行 GUI 探索，以适配异构系统并增强操作灵活性。摘要主要阐述问题与需求，未给出具体方法细节和实验评估结果。
Graphical User Interface (GUI) testing has been a significant topic in the software engineering community. Most existing GUI testing frameworks are intrusive and can only support some specific platforms, which are quite limited. With the development of distinct scenarios, diverse embedded systems or customized operating systems on different devices do not support existing intrusive GUI testing frameworks. Some approaches adopt robotic arms to replace the interface invoking of mobile apps under test and use computer vision technologies to identify GUI elements. However, some challenges remain unsolved with such approaches. First, existing approaches assume that GUI screens are fixed so that they cannot be adapted to diverse systems with different screen conditions. Second, existing approaches use XY-plane robotic arm system, which cannot flexibly simulate human testing operations. Third, existing approaches ignore the compatibility bugs of apps and only focus on the crash bugs. To sum up, a more practical approach is required for the non-intrusive scenario.
@inproceedings{paperbot11447,
title = {Practical Non-Intrusive GUI Exploration Testing with Visual-based Robotic Arms},
author = {Shengcheng Yu and Chunrong Fang and Mingzhe Du and Yuchen Ling and Zhenyu Chen and Zhendong Su},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3639161}
}