一句话：首个实证研究揭示 Java 项目中禁用测试的普遍性、生命周期、动机及其技术债影响。
论文针对测试维护中被 @Ignore 等注解临时禁用的失败测试，指出其可能演变为技术债，但缺乏实证证据。作者开发工具挖掘 15 个开源 Java 系统的 122K 次提交，识别出 3,111 次禁用测试变更，并进行生命周期分析与人工研究 349 个独特禁用测试的演化。结果显示，禁用变更比重命名、类型变更等常规测试重构常见 19%；41% 的禁用测试从未恢复以评估软件质量，且多数持续禁用数年；研究还归纳了禁用动机与相关技术债，最后提出对研究者和开发者的可操作启示。
Software testing is an essential software quality assurance practice. Testing helps expose faults earlier, allowing developers to repair the code and reduce future maintenance costs. However, repairing (i.e., making failing tests pass) may not always be done immediately. Bugs may require multiple rounds of repairs and even remain unfixed due to the difficulty of bug-fixing tasks. To help test maintenance, along with code comments, the majority of testing frameworks (e.g., JUnit and TestNG) have also introduced annotations such as @Ignore to disable failing tests temporarily. Although disabling tests may help alleviate maintenance difficulties, they may also introduce technical debt. With the faster release of applications in modern software development, disabling tests may become the salvation for many developers to meet project deliverables. In the end, disabled tests may become outdated and a source of technical debt, harming long-term maintenance. Despite its harmful implications, there is little empirical research evidence on the prevalence, evolution, and maintenance of disabling tests in practice. To fill this gap, we perform the first empirical study on test disabling practice. We develop a tool to mine 122K commits and detect 3,111 changes that disable tests from 15 open-source Java systems. Our main findings are: (1) Test disabling changes are 19% more common than regular test refactorings, such as renames and type changes. (2) Our life-cycle analysis shows that 41% of disabled tests are never brought back to evaluate software quality, and most disabled tests stay disabled for several years. (3)We unveil the motivations behind test disabling practice and the associated technical debt by manually studying evolutions of 349 unique disabled tests, achieving a 95% confidence level and a 5% confidence interval. Finally, we present some actionable implications for researchers and developers.
DOI 原文 ·
@inproceedings{paperbot6290,
title = {How disabled tests manifest in test maintainability challenges?},
author = {Dong Jae Kim and Bo Yang and Jinqiu Yang and Tse-Hsun (Peter) Chen},
booktitle = {FSE},
year = {2021},
doi = {10.1145/3468264.3468609}
}