GUI models encapsulate the desired visual appearance and interactive behaviors of applications, facilitating various downstream tasks like model-based testing (MBT). Manually constructing high-quality GUI models is not only labor-intensive and costly but also prone to errors, particularly as applications evolve and require frequent model updates. Existing automated approaches for GUI model generation heavily rely on reverse engineering, where the models are abstractions of the code. As a result, they are not suitable for MBT to test functional issues because they are consistent with the code. Meanwhile, valuable development artifacts such as UI/UX design documents, which reflect design intentions, are often overlooked. In this paper, a novel approach named DemGen is proposed to seek a unique pathway for GUI model generation. Leveraging design documents, DemGen employs computer vision pre-trained models in conjunction with a rule-based correction mechanism to identify GUI elements and their intended behaviors as defined in those documents. Subsequently, the identified content is transformed into a formal GUI model adhering to the IFML modeling language. Our evaluation, conducted in collaboration with an industry partner on commercial applications, demonstrates the effectiveness and efficiency of DemGen in GUI element recognition and GUI model generation. Moreover, we conducted a comparative analysis of manual, automated, and hybrid modeling techniques, assessing the usefulness of generated models on MBT tasks.

@inproceedings{paperbot8410,
  title = {Beyond Manual Modeling: Automating GUI Model Generation Using Design Documents},
  author = {Shaoheng Cao and Renyi Chen and Minxue Pan and Wenhua Yang and Xuandong Li},
  booktitle = {ASE},
  year = {2024},
  doi = {10.1145/3691620.3695032}
}