一句话：通过工业嵌入式系统案例，度量、归类并解释间歇性失败测试的根因。
间歇性失败测试的根因常难以定位；已有研究多聚焦开源软件的单元测试，工业嵌入式系统的系统级测试研究较少。本文以一套在运行的工业嵌入式系统及其系统级测试为对象，提出判定测试用例是否间歇失败的新度量，从逾五十万条测试判定中识别间歇性与一致性失败，并借助多来源信息分析根因。结果显示约1–3%的测试用例间歇失败，并归纳出九类相关因素。修复一致性失败测试通常比修复间歇性测试消除更多其他测试检测到的失败，且间歇性测试的修复成本更高；两类根因存在重叠。多数根因与开源软件相同，但嵌入式系统级测试中测试环境本身常导致间歇性。
Software testing is sometimes plagued with intermittently failing tests and finding the root causes of such failing tests is often difficult. This problem has been widely studied at the unit testing level for open source software, but there has been far less investigation at the system test level, particularly the testing of industrial embedded systems. This paper describes our investigation of the root causes of intermittently failing tests in the embedded systems domain, with the goal of better understanding, explaining and categorizing the underlying faults. The subject of our investigation is a currently-running industrial embedded system, along with the system level testing that was performed. We devised and used a novel metric for classifying test cases as intermittent. From more than a half million test verdicts, we identified intermittently and consistently failing tests, and identified their root causes using multiple sources. We found that about 1-3% of all test cases were intermittently failing. From analysis of the case study results and related work, we identified nine factors associated with test case intermittence. We found that a fix for a consistently failing test typically removed a larger number of failures detected by other tests than a fix for an intermittent test. We also found that more effort was usually needed to identify fixes for intermittent tests than for consistent tests. An overlap between root causes leading to intermittent and consistent tests was identified. Many root causes of intermittence are the same in industrial embedded systems and open source software. However, when comparing unit testing to system level testing, especially for embedded systems, we observed that the test environment itself is often the cause of intermittence.
@inproceedings{paperbot6072,
title = {Intermittently failing tests in the embedded systems domain},
author = {Per Erik Strandberg and Thomas J. Ostrand and Elaine J. Weyuker and Wasif Afzal and Daniel Sundmark},
booktitle = {ISSTA},
year = {2020},
doi = {10.1145/3395363.3397359}
}