一句话：提出在变异测试全流程中策略性重跑测试以管理 flaky 测试的技术，并在 PIT 中实现，大幅减少未知变异体。
变异测试广泛用于评估测试套件质量，但它隐含假设测试具有确定性：同一测试在相同被测代码上的覆盖与杀死变异体的结果应稳定。然而，flaky 测试会非确定性地改变结果，使变异分数不可靠。论文指出，重复执行时变异分数平均可相差四个百分点，且 9% 的变异体-测试对状态未知。为此，作者提出贯穿变异测试过程的 flaky 性管理技术，核心是基于策略性重跑测试，并修改开源变异测试工具 PIT 以实现这些技术。在 30 个项目上的评估表明，该方法将“未知”（flaky）变异体数量减少了 79.4%。
Mutation testing is widely used in research as a metric for evaluating the quality of test suites. Mutation testing runs the test suite on generated mutants (variants of the code under test), where a test suite kills a mutant if any of the tests fail when run on the mutant. Mutation testing implicitly assumes that tests exhibit deterministic behavior, in terms of their coverage and the outcome of a test (not) killing a certain mutant. Such an assumption does not hold in the presence of flaky tests, whose outcomes can non-deterministically differ even when run on the same code under test. Without reliable test outcomes, mutation testing can result in unreliable results, e.g., in our experiments, mutation scores vary by four percentage points on average between repeated executions, and 9% of mutant-test pairs have an unknown status. Many modern software projects suffer from flaky tests. We propose techniques that manage flakiness throughout the mutation testing process, largely based on strategically re-running tests. We implement our techniques by modifying the open-source mutation testing tool, PIT. Our evaluation on 30 projects shows that our techniques reduce the number of "unknown" (flaky) mutants by 79.4%.
DOI 原文 ·
@inproceedings{paperbot5392,
title = {Mitigating the effects of flaky tests on mutation testing},
author = {August Shi and Jonathan Bell and Darko Marinov},
booktitle = {ISSTA},
year = {2019},
doi = {10.1145/3293882.3330568}
}