一句话：提出受格式塔心理学启发的无监督方法，仅凭 GUI 像素图像推断控件感知分组。
GUI 中的控件并非彼此孤立，而是通过视觉线索形成 tab、menu、card、list 等更高层的感知单元；自动将 GUI 分割为控件的感知分组，是自动化 GUI 设计、实现与自动化任务的基础能力。然而，现有方法多依赖针对特定 GUI 实现与运行时信息的启发式规则，或依赖有监督机器学习，泛化性受限。心理学与生物视觉研究提出的格式塔知觉原则（连通性、相似性、邻近性、连续性等）从视觉线索层面刻画了人类如何对视觉元素分组，且与具体领域无关，已被广泛用于 GUI 内容组织。受此启发，本文提出一种新颖的、基于图像的无监督 GUI 控件感知分组推断方法：只需 GUI 像素图像，独立于 GUI 实现，且无需任何训练数据。在由 772 个移动应用与 20 个 UI 设计稿构成的 1,091 个 GUI 数据集上评估，该方法显著优于当前最先进的启发式基线，为改进 UI 相关软件工程任务创造了机会。
Graphical User Interface (GUI) is not merely a collection of individual and unrelated widgets, but rather partitions discrete widgets into groups by various visual cues, thus forming higher-order perceptual units such as tab, menu, card or list. The ability to automatically segment a GUI into perceptual groups of widgets constitutes a fundamental component of visual intelligence to automate GUI design, implementation and automation tasks. Although humans can partition a GUI into meaningful perceptual groups of widgets in a highly reliable way, perceptual grouping is still an open challenge for computational approaches. Existing methods rely on ad-hoc heuristics or supervised machine learning that is dependent on specific GUI implementations and runtime information. Research in psychology and biological vision has formulated a set of principles (i.e., Gestalt theory of perception) that describe how humans group elements in visual scenes based on visual cues like connectivity, similarity, proximity and continuity. These principles are domain-independent and have been widely adopted by practitioners to structure content on GUIs to improve aesthetic pleasantness and usability. Inspired by these principles, we present a novel unsupervised image-based method for inferring perceptual groups of GUI widgets. Our method requires only GUI pixel images, is independent of GUI implementation, and does not require any training data. The evaluation on a dataset of 1,091 GUIs collected from 772 mobile apps and 20 UI design mockups shows that our method significantly outperforms the state-of-the-art ad-hoc heuristics-based baseline. Our perceptual grouping method creates opportunities for improving UI-related software engineering tasks.
@inproceedings{paperbot7048,
title = {Psychologically-inspired, unsupervised inference of perceptual groups of GUI widgets from GUI images},
author = {Mulong Xie and Zhenchang Xing and Sidong Feng and Xiwei Xu and Liming Zhu and Chunyang Chen},
booktitle = {FSE},
year = {2022},
doi = {10.1145/3540250.3549138}
}