一句话：提出 Seenomaly，用无监督对抗自编码器“看”GUI 动画，检测其是否违反平台设计禁忌。
GUI 动画（如卡片移动、菜单滑入滑出、snackbar 显示）能提升移动应用的体验与可用性，但不应违反平台关于组件运动、内容显现与消失、层级与阴影变化的设计禁忌指南。现有静态代码分析、功能 GUI 测试和 GUI 图像比较都无法“看到”屏幕上的动画，因此无法支持此类 linting。作者将该问题形式化为多类 screencast 分类任务，但由于缺少标注的 GUI 动画数据，提出一种基于计算机视觉的无监督对抗自编码器：通过观看大量未标注的真实应用 GUI 动画并学习生成它们，将相似动画分组。作为同类首个工作，作者构建了合成与真实世界 GUI 动画数据集，并系统评估模型的学习能力、linting 有效性与实用性，同时指出未来挑战。
GUI animations, such as card movement, menu slide in/out, snackbar display, provide appealing user experience and enhance the usability of mobile applications. These GUI animations should not violate the platform's UI design guidelines (referred to as design-don't guideline in this work) regarding component motion and interaction, content appearing and disappearing, and elevation and shadow changes. However, none of existing static code analysis, functional GUI testing and GUI image comparison techniques can "see" the GUI animations on the scree, and thus they cannot support the linting of GUI animations against design-don't guidelines. In this work, we formulate this GUI animation linting problem as a multi-class screencast classification task, but we do not have sufficient labeled GUI animations to train the classifier. Instead, we propose an unsupervised, computer-vision based adversarial autoencoder to solve this linting problem. Our autoencoder learns to group similar GUI animations by "seeing" lots of unlabeled real-application GUI animations and learning to generate them. As the first work of its kind, we build the datasets of synthetic and real-world GUI animations. Through experiments on these datasets, we systematically investigate the learning capability of our model and its effectiveness and practicality for linting GUI animations, and identify the challenges in this linting problem for future work.
DOI 原文 ·
@inproceedings{paperbot5676,
title = {Seenomaly},
author = {Dehai Zhao and Zhenchang Xing and Chunyang Chen and Xiwei Xu and Liming Zhu and Guoqiang Li and Jinshui Wang},
booktitle = {ICSE},
year = {2020},
doi = {10.1145/3377811.3380411}
}