一句话：实证发现自动生成测试的 flakiness 至少与开发者测试相当，EvoSuite 抑制机制减少 71.7%，但成因分布不同。
测试的非确定性行为（flakiness）在开发者中常见且令人困扰，但以往研究几乎只关注开发者手工编写的测试，自动测试生成工具是否也会产生 flaky tests、其与手工测试有何差异仍不清楚。本文对 6,356 个 Java/Python 项目使用 EvoSuite 和 Pynguin 生成测试，并将每个测试执行 200 次以检测结果不一致；同时评估现有 flakiness 抑制机制。结果显示，生成测试中的 flakiness 至少与开发者测试同样普遍；EvoSuite 中现有抑制机制可减少 71.7% 的 flaky tests。与开发者测试相比，生成测试的 flakiness 更多由随机性而非网络和并发引起；抑制后残余 flakiness 主要归因于运行时优化和 EvoSuite 内部资源阈值，且显著不同于以往报告。作者发布数据集以支持工具维护、使用建议和后续研究。
Non-deterministic test behavior, or flakiness, is common and dreaded among developers. Researchers have studied the issue and proposed approaches to mitigate it. However, the vast majority of previous work has only considered developer-written tests. The prevalence and nature of flaky tests produced by test generation tools remain largely unknown. We ask whether such tools also produce flaky tests and how these differ from developer-written ones. Furthermore, we evaluate mechanisms that suppress flaky test generation. We sample 6 356 projects written in Java or Python. For each project, we generate tests using EvoSuite (Java) and Pynguin (Python), and execute each test 200 times, looking for inconsistent outcomes. Our results show that flakiness is at least as common in generated tests as in developer-written tests. Nevertheless, existing flakiness suppression mechanisms implemented in EvoSuite are effective in alleviating this issue (71.7 % fewer flaky tests). Compared to developer-written flaky tests, the causes of generated flaky tests are distributed differently. Their non-deterministic behavior is more frequently caused by randomness, rather than by networking and concurrency. Using flakiness suppression, the remaining flaky tests differ significantly from any flakiness previously reported, where most are attributable to runtime optimizations and EvoSuite-internal resource thresholds. These insights, with the accompanying dataset, can help maintainers to improve test generation tools, give recommendations for developers using these tools, and serve as a foundation for future research in test flakiness or test generation.
@inproceedings{paperbot11396,
title = {Do Automatic Test Generation Tools Generate Flaky Tests?},
author = {Martin Gruber and Muhammad Firhard Roslan and Owain Parry and Fabian Scharnböck and Phil McMinn and Gordon Fraser},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3608138}
}