一句话:揭示以变异体数量替代真实执行成本会威胁变异测试研究有效性,并复现经典研究证明影响显著。
变异测试研究常以变异体数量作为生成和执行变异体真实成本的代理度量,这可能威胁文献中科学结论的有效性。本文调查了 75 项工作,发现 54 项(72%)受此威胁影响。为量化该问题,作者对 10 个真实程序进行实证评估,结果显示:随机采样变异体的百分比与真实执行时间平均相差 44%,差异范围从 19% 到 91%;代理度量导致的误差与程序规模(ρ=0.74)和变异体数量(ρ=0.76)相关,说明越现实的程序问题越严重;在采样策略相关结论上,约 37% 的排序不一致,可能严重影响实验有效性。为检验实际影响,作者复现了一项关于选择性变异的重要研究,发现使用代理度量时的不确定结论在使用真实执行成本后变为明确结论。
Mutation testing research has often used the number of mutants as a surrogate measure for the true execution cost of generating and executing mutants. This poses a potential threat to the validity of the scientific findings reported in the literature. Out of 75 works surveyed in this paper, we found that 54 (72%) are vulnerable to this threat. To investigate the magnitude of the threat, we conducted an empirical evaluation using 10 real-world programs. The results reveal that: i) percentages of randomly sampled mutants differ from the true execution time, on average, by 44%, varying in difference from 19% to 91%; ii) errors arising from using the surrogate correlate with program size (ρ = 0.74) and number of mutants (ρ = 0.76), making the problem more pernicious for more realistic programs; iii) scientific findings concerning sampling strategies would have approximately 37% rank disagreement, indicating potentially dramatic impact on experiment validity. To investigate whether this threat matters in practice, we reproduced a seminal study on Selective Mutation (widely relied upon for more than two decades). The impact is stark: an inconclusive scientific finding using the surrogate is transformed to an unequivocal finding when using the true execution cost.