一句话：实证表明不稳定测试具有强故障发现能力，主张预测对象应从「不稳定测试」转向「不稳定测试执行」。
不稳定测试（flaky test）在同一程序版本的不同执行中时而通过时而失败，浪费开发者排查假告警的时间，已有大量预测方法试图识别它们。然而这些方法通常在单个发布版本的数据上评估，忽略了一个重要事实：某版本中偶发失败的测试，在另一个版本中可能真实失败并暴露缺陷，这种正确失败可能因此被忽视。本文通过跨版本实证分析表明该情形普遍存在：不稳定测试具有很强的故障揭示能力，揭示了超过三分之一的回归故障；在被测代码库中所有揭示故障的测试执行里，76.2% 来自现有预测方法判定为 flaky 的测试。这一发现对研究者与工业实践均构成警示，并促使未来研究转向预测「不稳定的测试执行」而非「不稳定的测试」本身。
Flaky tests are tests that pass and fail on different executions of the same version of a program under test. They waste valuable developer time by making developers investigate false alerts (flaky test failures). To deal with this issue, many prediction methods have been proposed. However, the utility of these methods remains unclear since they are typically evaluated based on single-release data, ignoring that in many cases tests that fail flakily in one release also correctly fail (indicating the presence of bugs) in some other, meaning that it is possible for subsequent correctly-failing cases to pass unnoticed. In this paper, we show that this situation is prevalent and can raise significant concerns for both researchers and practitioners. In particular, we show that flaky tests, tests that exhibit flaky behaviour at some point in time, have a strong fault-revealing capability, i.e., they reveal more than 1/3 of all encountered regression faults. We also show that 76.2%, of all test executions that reveal faults in the codebase under test are made by tests that are classified as flaky by existing prediction methods. Overall, our findings motivate the need for future research to focus on predicting flaky test executions instead of flaky tests.CCS CONCEPTS• Software and its engineering → Software testing and debugging.
@inproceedings{HabenHMHPCT24,
title = {The Importance of Accounting for Execution Failures when Predicting Test Flakiness},
author = {Guillaume Haben and Sarra Habchi and John Micco and Mark Harman and Mike Papadakis and Maxime Cordy and Yves Le Traon},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695261}
}