一句话：在 68 个工业级 Android 应用上实证比较主流 UI 测试生成工具的覆盖率与缺陷检测能力，并给出工具组合建议。
UI 测试是保障移动应用质量的重要手段，已有大量面向 Android 的测试生成工具，但以往评测多基于相对简单的开源应用，缺乏在真实工业应用上对工具有效性与易用性的直接对比。本文选取 68 个广泛使用的工业级 Android 应用，对当前最先进（state-of-the-art）与工业界常用（state-of-the-practice）的测试生成工具进行系统实证研究，直接比较它们在代码覆盖率与故障检测能力上的表现。结果显示：Google 的工业工具 Monkey 在可获得方法覆盖率数据的 41 个应用中于 22 个取得最高方法覆盖率，并在全部 68 个应用中于 35 个取得最高 Activity 覆盖率；而学术工具 Stoat 在 23 个应用中触发了最多的唯一崩溃。基于实验分析，作者进一步给出组合使用不同工具以提升效果的建议，并报告了在工业应用中落地这些工具的经验，为 Android UI 测试生成工具的改进方向提供了洞见。
User Interface (UI) testing is a popular approach to ensure the quality of mobile apps. Numerous test generation tools have been developed to support UI testing on mobile apps, especially for Android apps. Previous work evaluates and compares different test generation tools using only relatively simple open-source apps, while real-world industrial apps tend to have more complex functionalities and implementations. There is no direct comparison among test generation tools with regard to effectiveness and ease-of-use on these industrial apps. To address such limitation, we study existing state-of-the-art or state-of-the-practice test generation tools on 68 widely-used industrial apps. We directly compare the tools with regard to code coverage and fault-detection ability. According to our results, Monkey, a state-of-the-practice tool from Google, achieves the highest method coverage on 22 of 41 apps whose method coverage data can be obtained. Of all 68 apps under study, Monkey also achieves the highest activity coverage on 35 apps, while Stoat, a state-of-the-art tool, is able to trigger the highest number of unique crashes on 23 apps. By analyzing the experimental results, we provide suggestions for combining different test generation tools to achieve better performance. We also report our experience in applying these tools to industrial apps under study. Our study results give insights on how Android UI test generation tools could be improved to better handle complex industrial apps.
DOI 原文 ·
@inproceedings{paperbot4716,
title = {An empirical study of Android test generation tools in industrial cases},
author = {Wenyu Wang and Dengfeng Li and Wei Yang and Yurui Cao and Zhenwen Zhang and Yuetang Deng and Tao Xie},
booktitle = {ASE},
year = {2018},
doi = {10.1145/3238147.3240465}
}