一句话：提出 Minuku，通过微调视觉语言模型模拟人类常识，在小规模数据下检测移动应用 UI 显示问题。
移动应用 UI 显示问题（控件遮挡、元素缺失、屏幕溢出等）已成为用户投诉的重要来源，但现有自动化检测工具依赖大量高质量训练数据，工业应用成本高。本文提出 Minuku：利用显示问题可被人直觉识别这一特点，将其抽象为对人类 UI 外观常识预期的违反，从而通过常识模拟降低数据需求。具体地，作者微调视觉语言模型（VLM），使其学习何为预期显示并推理潜在显示问题。评估包括消融实验，并在某大型电商应用提供商真实部署，结果表明 Minuku 能检测出 40 个此前未知的 UI 显示问题，并显著减少人工投入。
User interface (UI) display issues, such as widgets occlusion, missing elements, and screen overflow, are emerging as a non-negligible source of user complaints in commercial mobile apps. However, existing automated testing tools typically rely on a vast amount of high-quality training data, making them cost-ineffective for industrial practice. Given that display issues are intuitively recognizable by humans, their diverse appearances can be abstracted by the violation of human commonsense expectations of UI appearance. Therefore, this paper proposes to reduce data requirements in display issue detection through commonsense simulation. Although leveraging large vision-language models (VLMs) to replicate human visual ability looks straightforward, off-the-shelf VLMs lack task-specific knowledge of UI designs and display correctness. To address this, we fine-tune a VLM to learn what constitutes an expected display and to reason potential display issues. This approach is termed as Minuku, an industrial data-efficient UI display issue detector. We evaluate the design effectiveness of Minuku via a set of ablation experiments. Moreover, real-world deployments in one of the largest E-commerce app providers further demonstrate that Minuku can effectively detect 40 previously unknown UI display issues and significantly reduce manual effort in industrial settings.
DOI 原文 ·
@inproceedings{paperbot9227,
title = {Minuku: Detecting Diverse Display Issues in Mobile Apps with Small-scale Dataset},
author = {Yongxiang Hu and Ke Liu and Hailiang Jin and Shiyu Guo and Juxing Yuan and Xin Wang and Yangfan Zhou},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00270}
}