一句话：澄清测试集规模在充分性准则与故障检测关系中的角色，并给出公平比较准则的方法论与实证评估。
本文重新审视故障检测、测试充分性准则与测试集规模三者之间的复杂关系。作者指出，以往研究对该关系存在相互矛盾的结论，并明确论证测试集规模既不是先前所认为的混淆变量，也不是应当被实验操纵的自变量。论文首先综述并化解了这些矛盾分析，随后阐明既有工作的实验设计，讨论其中的概念与统计问题，并给出面向未来研究的指导准则。在此基础上，作者提出一种在考虑测试集规模的前提下、不通过不切实际分层来直接操纵规模的方法论，从而实现对测试充分性准则的公平比较。最后，论文开展实证评估，比较基于覆盖率、基于变异与随机测试的有效性，并进一步提出概率耦合（probabilistic coupling）方法，用于评估给定故障下测试目标集合的代表性，以及近似充分测试集的故障检测概率。
The research community has long recognized a complex interrelationship between fault detection, test adequacy criteria, and test set size. However, there is substantial confusion about whether and how to experimentally control for test set size when assessing how well an adequacy criterion is correlated with fault detection and when comparing test adequacy criteria. Resolving the confusion, this paper makes the following contributions: (1) A review of contradictory analyses of the relationships between fault detection, test adequacy criteria, and test set size. Specifically, this paper addresses the supposed contradiction of prior work and explains why test set size is neither a confounding variable, as previously suggested, nor an independent variable that should be experimentally manipulated. (2) An explication and discussion of the experimental designs of prior work, together with a discussion of conceptual and statistical problems, as well as specific guidelines for future work. (3) A methodology for comparing test adequacy criteria on an equal basis, which accounts for test set size without directly manipulating it through unrealistic stratification. (4) An empirical evaluation that compares the effectiveness of coverage-based testing, mutation-based testing, and random testing. Additionally, this paper proposes probabilistic coupling, a methodology for assessing the representativeness of a set of test goals for a given fault and for approximating the fault-detection probability of adequate test sets.
@inproceedings{paperbot5969,
title = {Revisiting the relationship between fault detection, test adequacy criteria, and test set size},
author = {Yiqun T. Chen and Rahul Gopinath and Anita Tadakamalla and Michael D. Ernst and Reid Holmes and Gordon Fraser and Paul Ammann and René Just},
booktitle = {ASE},
year = {2020},
doi = {10.1145/3324884.3416667}
}