一句话：提出UIED工具，融合传统CV与深度学习，实现精确GUI元素检测并支持结果编辑与设计文件导出。
GUI元素检测是GUI自动化、测试、逆向工程和设计还原的第一步，但复杂多样的界面图像使准确检测仍具挑战。本文实现UIED工具包，为使用者提供简单易用的精确GUI元素检测平台。UIED集成传统计算机视觉方法与深度学习模型以应对不同复杂度的GUI图像，并配备定制的GUI元素检测方法，以获得当前最优的检测结果。工具支持在交互式面板中修改和编辑检测结果，还能将检测到的UI元素导出为主流UI设计工具（如Sketch、Photoshop）可继续编辑的设计文件。评估表明UIED能实现准确检测，并对下游工作有用。
Graphical User Interface (GUI) elements detection is critical for many GUI automation and GUI testing tasks. Acquiring the accurate positions and classes of GUI elements is also the very first step to conduct GUI reverse engineering or perform GUI testing. In this paper, we implement a User Iterface Element Detection (UIED), a toolkit designed to provide user with a simple and easy-to-use platform to achieve accurate GUI element detection. UIED integrates multiple detection methods including old-fashioned computer vision (CV) approaches and deep learning models to handle diverse and complicated GUI images. Besides, it equips with a novel customized GUI element detection methods to produce state-of-the-art detection results. Our tool enables the user to change and edit the detection result in an interactive dashboard. Finally, it exports the detected UI elements in the GUI image to design files that can be further edited in popular UI design tools such as Sketch and Photoshop. UIED is evaluated to be capable of accurate detection and useful for downstream works.
DOI 原文 ·
@inproceedings{paperbot5802,
title = {UIED: a hybrid tool for GUI element detection},
author = {Mulong Xie and Sidong Feng and Zhenchang Xing and Jieshan Chen and Chunyang Chen},
booktitle = {FSE},
year = {2020},
doi = {10.1145/3368089.3417940}
}