一句话：通过在测试执行环境中注入竞争 CPU/内存资源的噪声，Shaker 能比 ReRun 更快发现更多 flaky 测试。
flaky 测试指在相同源码与测试代码版本下时而通过时而失败的测试，会浪费大量测试时间与资源。工业界主流检测方法是 ReRun，即对失败测试反复重跑以寻找输出不一致，但该策略在时间与计算资源上开销极高，尤其对每日执行成千上万测试的场景。本文提出开源工具 Shaker，通过在时间受限测试的执行环境中加入与测试竞争 CPU 或内存资源的压力任务，以噪声触发不稳定行为。Shaker 可作为 GitHub Actions workflow 无缝集成到任意 GitHub 项目，也可通过命令行接口使用。评估表明，Shaker 比 ReRun 发现更多 flaky 测试且需要更少重执行次数；此外还发现了数十个 ReRun 即使重执行 50 次也未检出的新 flaky 测试。
A test case that intermittently passes or fails when performed under the same version of source code and test code is said to be flaky. The presence of flaky tests wastes testing time and effort. The most popular approach in industry to detect flakiness is ReRun. The idea behind ReRun is very simple: failing test cases are re-executed many times looking for inconsistencies in the output. Despite its simplicity, the ReRun strategy is very expensive both in terms of time and in terms of computational resources. This is particularly true for contexts where thousands of test cases are performed on a daily basis. Reducing the rerunning overhead is, thus, of utmost importance. This paper presents SHAKER, an open-source tool for detecting flakiness in time-constrained tests by adding noise in the execution environment. The main idea behind SHAKER is to add stressing tasks that compete with the test execution for the use of resources (CPU or memory). SHAKER is available as a GitHub Actions workflow that can be seamlessly integrated with any GitHub project. Alternatively, SHAKER can also be used via its provided Command Line Interface. In our evaluation, SHAKER was able to discover more flaky tests than ReRun and in a faster way (less re-executions); besides, our approach revealed tens of new flaky tests that went undetected by ReRun even after 50 re-executions. Thanks to its flexibility and ease of use, we believe that SHAKER can be useful for both practitioners and researchers.Demo video: https://youtu.be/7-aiQwOb4rAShaker website: https://star-rg.github.io/shaker
DOI 原文 ·
@inproceedings{paperbot6545,
title = {Shaker: a Tool for Detecting More Flaky Tests Faster},
author = {Marcello Cordeiro and Denini Silva and Leopoldo Teixeira and Breno Miranda and Marcelo d'Amorim},
booktitle = {ASE},
year = {2021},
doi = {10.1109/ase51524.2021.9678918}
}