一句话：提出 OwlEye，用深度学习建模 GUI 截图，检测并定位 UI 显示问题以辅助修复。
GUI 复杂度给实现带来挑战；众测报告显示，文本重叠、屏幕模糊、图片缺失等显示问题常因软硬件兼容性在不同设备渲染时出现，影响可用性与用户体验。论文提出 OwlEye，基于深度学习建模 GUI 截图的视觉信息，不仅能检测存在显示问题的 GUI，还能定位问题区域以指导开发者修复。作者构建了含 4,470 张带 UI 显示问题截图的大规模标注数据集，并提出基于启发式和基于 GAN 的数据增强方法来提升性能。评估表明，OwlEye 检测 UI 显示问题达到 85% 精确率和 84% 召回率，问题定位准确率为 90%。
GUI complexity posts a great challenge to the GUI implementation. According to our pilot study of crowdtesting bug reports, display issues such as text overlap, blurred screen, missing image always occur during GUI rendering on difference devices due to the software or hardware compatibility. They negatively influence the app usability, resulting in poor user experience. To detect these issues, we propose a novel approach, OwlEye, based on deep learning for modelling visual information of the GUI screenshot. Therefore, OwlEye can detect GUIs with display issues and also locate the detailed region of the issue in the given GUI for guiding developers to fix the bug. We manually construct a large-scale labelled dataset with 4,470 GUI screenshots with UI display issues. We develop a heuristics-based data augmentation method and a GAN-based data augmentation method for boosting the performance of our OwlEye. At present, the evaluation demonstrates that our OwlEye can achieve 85% precision and 84% recall in detecting UI display issues, and 90% accuracy in localizing these issues.
DOI 原文 ·
@inproceedings{paperbot5992,
title = {Discovering UI display issues with visual understanding},
author = {Zhe Liu},
booktitle = {ASE},
year = {2020},
doi = {10.1145/3324884.3418917}
}