paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Cost measures matter for mutation testing study validity

成本度量对变异测试研究有效性至关重要
FSE Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering2020匹配度 1/10 · 该论文研究变异测试中执行成本度量对研究有效性的影响,属于软件测试与实证方法学,与静态分析、LLM/Agent、浮点程序分析或贝叶斯程序分析均无直接关系。虽然问题驱动且关注有效性威胁,但不在用户研究画像的核心方向内。引用 11 inbox
Giovani Guizzo, Federica Sarro, Mark Harman
crossref: 2026-09-12

AI 速览

一句话:揭示以变异体数量替代真实执行成本会威胁变异测试研究有效性,并复现经典研究证明影响显著。

变异测试研究常以变异体数量作为生成和执行变异体真实成本的代理度量,这可能威胁文献中科学结论的有效性。本文调查了 75 项工作,发现 54 项(72%)受此威胁影响。为量化该问题,作者对 10 个真实程序进行实证评估,结果显示:随机采样变异体的百分比与真实执行时间平均相差 44%,差异范围从 19% 到 91%;代理度量导致的误差与程序规模(ρ=0.74)和变异体数量(ρ=0.76)相关,说明越现实的程序问题越严重;在采样策略相关结论上,约 37% 的排序不一致,可能严重影响实验有效性。为检验实际影响,作者复现了一项关于选择性变异的重要研究,发现使用代理度量时的不确定结论在使用真实执行成本后变为明确结论。

问题
变异测试研究常将变异体数量作为真实执行成本的代理度量,这可能削弱研究结论的有效性。
方法
调查 75 篇文献评估该威胁的普遍性,并在 10 个真实程序上比较代理度量与真实执行成本,同时复现选择性变异经典研究。
评估
通过文献调查、真实程序上的实证评估以及经典研究的复现实验,量化代理度量与真实执行成本的偏差及其对科学结论的影响。
原文摘要(Abstract)

Mutation testing research has often used the number of mutants as a surrogate measure for the true execution cost of generating and executing mutants. This poses a potential threat to the validity of the scientific findings reported in the literature. Out of 75 works surveyed in this paper, we found that 54 (72%) are vulnerable to this threat. To investigate the magnitude of the threat, we conducted an empirical evaluation using 10 real-world programs. The results reveal that: i) percentages of randomly sampled mutants differ from the true execution time, on average, by 44%, varying in difference from 19% to 91%; ii) errors arising from using the surrogate correlate with program size (ρ = 0.74) and number of mutants (ρ = 0.76), making the problem more pernicious for more realistic programs; iii) scientific findings concerning sampling strategies would have approximately 37% rank disagreement, indicating potentially dramatic impact on experiment validity. To investigate whether this threat matters in practice, we reproduced a seminal study on Selective Mutation (widely relied upon for more than two decades). The impact is stark: an inconclusive scientific finding using the surrogate is transformed to an unequivocal finding when using the true execution cost.

链接与引用

DOI 原文 · PDF(开放获取) ·

BibTeX
@inproceedings{paperbot5758,
  title = {Cost measures matter for mutation testing study validity},
  author = {Giovani Guizzo and Federica Sarro and Mark Harman},
  booktitle = {FSE},
  year = {2020},
  doi = {10.1145/3368089.3409742}
}