一句话：提出 EGFE，用多模态 Transformer 与 UI 序列预测端到端分组碎片化 UI 元素，改善设计到代码生成。
将 UI 设计原型自动转为代码可加速应用开发与 GUI 迭代，但缺乏严格规范时，UI 组件常由碎片化元素构成；分组这些元素能提升生成代码的可读性与可维护性。现有两阶段方法依赖手工规则，受视觉重叠和微小元素影响，效果不理想。本文提出 EGFE，将碎片化元素分组建模为 UI 序列预测，构建 Transformer 编码器，通过多模态表示学习建模元素间关系，实现端到端分组。在 4606 个专业设计师 UI 原型数据集上，编辑距离阈值 4 时，精确率、召回率和 F1 分别相对提升 29.75%、31.07% 和 30.39%，优于 SOTA；实证研究验证了其对生成前端代码的改进及真实软件工程应用价值。
When translating UI design prototypes to code in industry, automatically generating code from design prototypes can expedite the development of applications and GUI iterations. However, in design prototypes without strict design specifications, UI components may be composed of fragmented elements. Grouping these fragmented elements can greatly improve the readability and maintainability of the generated code. Current methods employ a two-stage strategy that introduces hand-crafted rules to group fragmented elements. Unfortunately, the performance of these methods is not satisfying due to visually overlapped and tiny UI elements. In this study, we propose EGFE, a novel method for automatically End-to-end Grouping Fragmented Elements via UI sequence prediction. To facilitate the UI understanding, we innovatively construct a Transformer encoder to model the relationship between the UI elements with multi-modal representation learning. The evaluation on a dataset of 4606 UI prototypes collected from professional UI designers shows that our method outperforms the state-of-the-art baselines in the precision (by 29.75%), recall (by 31.07%), and F1-score (by 30.39%) at edit distance threshold of 4. In addition, we conduct an empirical study to assess the improvement of the generated front-end code. The results demonstrate the effectiveness of our method on a real software engineering application. Our end-to-end fragmented elements grouping method creates opportunities for improving UI-related software engineering tasks.
@inproceedings{paperbot11476,
title = {EGFE: End-to-end Grouping of Fragmented Elements in UI Designs with Multimodal Learning},
author = {Liuqing Chen and Yunnong Chen and Shuhong Xiao and Yaxuan Song and Lingyun Sun and Yankun Zhen and Tingting Zhou and Yanfang Chang},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3623313}
}