一句话：提出用机器学习从屏幕截图识别 GUI 控件，并以此指导随机 GUI 测试，显著提升分支覆盖率。
GUI 自动化测试工具通常依赖操作系统或 GUI 框架特定知识来与程序交互，但系统更新和框架多样性会使这些信息失效，导致工具难以跨环境支持。本文提出一种通过机器学习从屏幕截图中自动识别 GUI 控件来改进 GUI 测试的技术。为获得训练数据，作者自动生成随机 GUI 并从中提取控件信息。得到的模型仅凭屏幕截图即可导出控件信息，从而在不被现有工具支持的环境中为 GUI 测试提供引导。实验在 20 个应用上将该方法与常规随机测试对比，其中 18 个应用取得了显著更高的分支覆盖率，平均提升 42.5%。
Graphical User Interfaces (GUIs) are amongst the most common user interfaces, enabling interactions with applications through mouse movements and key presses. Tools for automated testing of programs through their GUI exist, however they usually rely on operating system or framework specific knowledge to interact with an application. Due to frequent operating system updates, which can remove required information, and a large variety of different GUI frameworks using unique underlying data structures, such tools rapidly become obsolete, Consequently, for an automated GUI test generation tool, supporting many frameworks and operating systems is impractical. We propose a technique for improving GUI testing by automatically identifying GUI widgets in screen shots using machine learning techniques. As training data, we generate randomized GUIs to automatically extract widget information. The resulting model provides guidance to GUI testing tools in environments not currently supported by deriving GUI widget information from screen shots only. In our experiments, we found that identifying GUI widgets in screen shots and using this information to guide random testing achieved a significantly higher branch coverage in 18 of 20 applications, with an average increase of 42.5% when compared to conventional random testing.
DOI 原文 ·
@inproceedings{paperbot5345,
title = {Improving random GUI testing with image-based widget detection},
author = {Thomas D. White and Gordon Fraser and Guy J. Brown},
booktitle = {ISSTA},
year = {2019},
doi = {10.1145/3293882.3330551}
}