一句话：提出基于自动机的轨迹分析，为 Android GUI 测试工具诊断漏报缺陷提供线索，并增强 Themis 基准。
在 Android GUI 测试中，用已知缺陷评测测试工具难以给出漏报缺陷的线索，导致大量临时人工分析。本文提出基于自动机的轨迹分析：将缺陷建模为刻画触发轨迹的有限自动机，把测试工具生成的冗长事件轨迹与该自动机匹配，并以三个基于自动机的覆盖值给出诊断线索。作者据此增强 Android 基准套件 Themis，形成 Themis+。在九个先进 GUI 测试工具上评估并与开发者合作表明，Themis+ 的线索与人工分析结果一致或有用，发现了此前未知的工具弱点；两个工业工具据此快速优化并提升了缺陷发现能力，开发者反馈积极。
Benchmarking software testing tools against known bugs is a classic approach to evaluating the tools’ bug finding abilities. However, this approach is difficult to give some clues on the tool-missed bugs to aid diagnosing the testing tools. As a result, heavy and ad hoc manual analysis is needed. In this work, in the setting of GUI testing for Android apps, we introduce an automata-based trace analysis approach to tackling the key challenge of manual analysis, i.e., how to analyze the lengthy event traces generated by a testing tool against a missed bug to find the clues. Our key idea is that, we model a bug in the form of a finite automaton which captures its bug-triggering traces; and match the event traces generated by the testing tool (which misses this bug) against this automaton to obtain the clues. Specifically, the clues are presented in the form of three designated automata-based coverage values. We apply our approach to enhance Themis, a representative benchmark suite for Android, to aid diagnosing GUI testing tools. Our extensive evaluation on nine state-of-the-art GUI testing tools and the involvement with several tool developers shows that our approach is feasible and useful. Our approach enables Themis+ (the enhanced benchmark suite) to provide the clues on the tool-missed bugs, and all the Themis+’s clues are identical or useful, compared to the manual analysis results of tool developers. Moreover, the clues have helped find several tool weaknesses, which were unknown or unclear before. Based on the clues, two actively-developing industrial testing tools in our study have quickly made several optimizations and demonstrated their improved bug finding abilities. All the tool developers give positive feedback on the usefulness and usability of Themis+’s clues. Themis+ is available at https://github.com/DDroid-Android/home.
@inproceedings{MaHH0WLP023,
title = {Automata-Based Trace Analysis for Aiding Diagnosing GUI Testing Tools for Android},
author = {Enze Ma and Shan Huang and Weigang He and Ting Su and Jue Wang and Huiyu Liu and Geguang Pu and Zhendong Su},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616361}
}