一句话：提出基于动态分析推断 GUI 事件竞态并选择性延迟相关事件的方法，可快速可靠复现 Android 时序相关 flaky 测试。
不稳定测试（flaky tests）在同一代码版本上时而通过、时而失败，常误导开发者调试不存在的缺陷，而传统多次重跑复现方式耗时且可靠性低。本文聚焦 Android 应用中最常见的时序相关 GUI flaky 测试，其核心见解是这类不确定性常源于 GUI 数据上的事件竞态。给定失败栈迹，技术通过动态分析推断可能导致该失败的事件竞态，并仅选择性地延迟参与竞态的相关事件，从而以最少测试运行次数复现失败。作者在从 22 个广泛使用的 Android 应用收集的 80 个时序相关 flaky 测试上进行实验，成功复现其中 73 个，平均仅需 1.71 次测试运行；并且能够连续 20 次稳定复现同一失败，表现出很高可靠性。
Flaky tests hinder the development process by exhibiting uncertain behavior in regression testing. A flaky test may pass in some runs and fail in others while running on the same code version. The non-deterministic outcome frequently misleads the developers into debugging non-existent faults in the code. To effectively debug the flaky tests, developers need to reproduce them. The industry de facto to reproduce flaky tests is to rerun them multiple times. However, rerunning a flaky test numerous times is time and resource-consuming. This work presents a technique for rapidly and reliably reproducing timing-dependent GUI flaky tests, acknowledged as the most common type of flaky tests in Android apps. Our insight is that flakiness in such tests often stems from event racing on GUI data. Given stack traces of a failure, our technique employs dynamic analysis to infer event races likely leading to the failure and reproduces it by selectively delaying only relevant events involved in these races. Thus, our technique can efficiently reproduce a failure within minimal test runs. The experiments conducted on 80 timing-dependent flaky tests collected from 22 widely-used Android apps show our technique is efficient in flaky test failure reproduction. Out of the 80 flaky tests, our technique could successfully reproduce 73 within 1.71 test runs on average. Notably, it exhibited extremely high reliability by consistently reproducing the failure for 20 runs.
DOI 原文 ·
@inproceedings{paperbot8480,
title = {Reproducing Timing-Dependent GUI Flaky Tests in Android Apps via a Single Event Delay},
author = {Xiaobao Cai and Zhen Dong and Yongjiang Wang and Abhishek Tiwari and Xin Peng},
booktitle = {ISSTA},
year = {2024},
doi = {10.1145/3650212.3680377}
}