一句话：对 7 个 Node.js 应用的 87 个不稳定测试开展实证研究，归纳五类非确定性根因，指出现有事件竞争检测方法不足。
回归测试对评估 Node.js 应用质量很重要，但 Node.js 框架中的非确定性执行会使测试在同一代码版本上间歇性通过或失败，即 flaky tests。这类测试导致结果不可靠，并浪费开发者时间调试非目标应用的缺陷。本文对 7 个流行 Node.js 应用中的 87 个 flaky tests 进行实证研究，分析其非确定性来源。结果发现，非确定性来源广泛，包括事件触发顺序、函数调用、进程/线程调度、异步任务执行以及事件触发数据。研究还表明，现有事件竞争检测方法不足以检测 flaky tests，未来可针对不同类别的非确定性设计专门检测方法。
Regression testing is an important means of assessing the quality of Node.js applications. However, non-deterministic executions inside Node.js framework could make test cases intermittently pass or fail on the same version of code, which are called flaky tests. Flaky tests can cause unreliable test results, and make developers waste a significant amount of time debugging the bugs that do not belong to the target application. In this paper, we conduct an empirical study on 87 flaky tests from 7 popular Node.js applications, and analyze the non-determinism that causes these flaky tests. Through this study, there is a wide range of non-determinism to cause flaky tests, including non-deterministic event triggering order, non-deterministic function calls, non-deterministic process/thread scheduling order, non-deterministic execution of asynchronous tasks and non-deterministic event triggering data. The result reveals that, existing approaches on event race detection are not sufficient for flaky test detection. In future, researchers can design flaky test detection approaches targeted at different categories of non-determinism.
@inproceedings{ChangLWGCWLCH23,
title = {Characterizing Flaky Tests in Node.js Applications},
author = {Xiaoning Chang and Zheheng Liang and Guoquan Wu and Yu Gao and Wei Chen and Jun Wei and Zhenyue Long and Lei Cui and Tao Huang},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00025}
}