一句话：提出神经机器翻译器，无需手工规则即可将 UI 设计图像自动转换为 GUI 骨架。
论文关注从 UI 设计图生成 GUI 骨架这一问题。开发者通常需人工理解界面元素及其空间布局，再映射为合适的 GUI 组件和组合，过程繁琐且依赖经验。现有工具依赖启发式视觉理解与手写生成规则，难以适应多样的 UI 设计和复杂的骨架结构。作者提出一个神经机器翻译器，在统一神经网络框架中融合计算机视觉与机器翻译：从 UI 图像中提取视觉特征，编码其空间布局，并生成 GUI 骨架，无需人工规则。为训练模型，作者开发了自动化 GUI 探索方法，从真实应用中收集大规模 UI 数据。实验评估了方法的准确率、泛化性和实用性。
A GUI skeleton is the starting point for implementing a UI design image. To obtain a GUI skeleton from a UI design image, developers have to visually understand UI elements and their spatial layout in the image, and then translate this understanding into proper GUI components and their compositions. Automating this visual understanding and translation would be beneficial for bootstraping mobile GUI implementation, but it is a challenging task due to the diversity of UI designs and the complexity of GUI skeletons to generate. Existing tools are rigid as they depend on heuristically-designed visual understanding and GUI generation rules. In this paper, we present a neural machine translator that combines recent advances in computer vision and machine translation for translating a UI design image into a GUI skeleton. Our translator learns to extract visual features in UI images, encode these features' spatial layouts, and generate GUI skeletons in a unified neural network framework, without requiring manual rule development. For training our translator, we develop an automated GUI exploration method to automatically collect large-scale UI data from real-world applications. We carry out extensive experiments to evaluate the accuracy, generality and usefulness of our approach.
DOI 原文 ·
@inproceedings{paperbot4489,
title = {From UI design image to GUI skeleton},
author = {Chunyang Chen and Ting Su and Guozhu Meng and Zhenchang Xing and Yang Liu},
booktitle = {ICSE},
year = {2018},
doi = {10.1145/3180155.3180240}
}