一句话：首个研究自动化“缺陷 UI 定位”的可行性，比较文本与多模态深度学习技术，并证明其可提升代码定位。
移动应用由 UI 驱动，缺陷通常通过 UI 表现出来，因此识别有缺陷的 UI 屏幕与 UI 组件（Buggy UI Localization）对定位缺陷行为并最终修复缺陷十分重要。然而该任务颇具挑战：开发者必须推理质量往往很低的缺陷描述，以及 UI 屏幕的视觉或代码表示。本文首次系统研究该任务自动化的可行性，评估了一种纯文本方法、两种多模态深度学习方法和一种无监督文本方法，并在 UI 屏幕与 UI 组件两个粒度上进行定位评估。结果显示不同表示各有所长：融合视觉信息的模型在 UI 屏幕定位上表现更好，而基于文本屏幕信息的模型在 UI 组件定位上表现更好，说明需要结合两者优势的定位方法。此外，作者研究了缺陷 UI 定位能否改进传统缺陷代码定位，发现引入定位到的缺陷 UI 后 Hits@10 提升 9%–12%。
Bug report management is a costly software maintenance process comprised of several challenging tasks. Given the UI-driven nature of mobile apps, bugs typically manifest through the UI, hence the identification of buggy UI screens and UI components (Buggy UI Localization) is important to localizing the buggy behavior and eventually fixing it. However, this task is challenging as developers must reason about bug descriptions (which are often low-quality), and the visual or code-based representations of UI screens. This paper is the first to investigate the feasibility of automating the task of Buggy UI Localization through a comprehensive study that evaluates the capabilities of one textual and two multi-modal deep learning (DL) techniques and one textual unsupervised technique. We evaluate such techniques at two levels of granularity, Buggy UI Screen and UI Component localization. Our results illustrate the individual strengths of models that make use of different representations, wherein models that incorporate visual information perform better for UI screen localization, and models that operate on textual screen information perform better for UI component localization – highlighting the need for a localization approach that blends the benefits of both types of techniques. Furthermore, we study whether Buggy UI Localization can improve traditional buggy code localization, and find that incorporating localized buggy UIs leads to improvements of 9%-12% in Hits@10.
@inproceedings{paperbot8517,
title = {Toward the Automated Localization of Buggy Mobile App UIs from Bug Descriptions},
author = {Antu Saha and Yang Song and Junayed Mahmud and Ying Zhou and Kevin Moran and Oscar Chaparro},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680357}
}