一句话：通过大规模实证对比七种 GUI 元素检测方法，提出融合传统粗到细策略与深度学习文本检测的新方法。
该文研究 GUI 图像中的元素检测这一领域特定目标检测任务，它可支撑 GUI 动画、测试、搜索和代码生成等软件工程任务。作者首次在超过 5 万张 GUI 图像上对七种代表性检测方法（传统图像处理与深度学习模型）进行大规模实证研究，分析其能力、局限与有效设计。基于此，作者提出一种面向 GUI 的新传统方法，用于非文本元素检测，采用新颖的自顶向下由粗到细策略，并与成熟的深度学习模型结合以检测 GUI 文本。在 25,000 张 GUI 图像上的评估表明，该方法显著提升了 GUI 元素检测的当前最佳性能。
Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific old-fashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection.Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.
@inproceedings{paperbot5828,
title = {Object detection for graphical user interface: old fashioned or deep learning or a combination?},
author = {Jieshan Chen and Mulong Xie and Zhenchang Xing and Chunyang Chen and Xiwei Xu and Liming Zhu and Guoqiang Li},
booktitle = {FSE},
year = {2020},
doi = {10.1145/3368089.3409691}
}