一句话：实证表明 Firefox 构建中失败、flaky 及高频失败测试与用户崩溃报告数大幅增加相关，忽略测试失败会显著提升发布版崩溃。
持续集成中失败测试常被忽略并随构建发布。本文实证研究 Firefox 构建的测试失败与用户崩溃报告数的关系。作者将 Beta 与 Production 构建按测试全通过、存在失败测试、flaky 测试、高频失败测试分组，比较各组崩溃报告数中位数。结果显示：全通过构建仅 2 个崩溃报告；存在失败测试的构建在 Beta 和 Production 上分别达 508 和 291；flaky 测试为 514 和 234；高频失败测试为 585 和 780，对 Production 影响大于 Beta，约为普通失败测试的 2.7 倍。研究表明忽略测试失败与用户报告崩溃数大幅上升相关。
Testing is an integral part of release engineering and continuous integration. In theory, a failed test on a build indicates a problem that should be fixed and the build should not be released. In practice, tests decay and developers often release builds, ignoring failing tests. In this paper, we studying the link between builds with failing tests and the number of crash reports on the Firefox webbrowser. Builds with all tests passing have a median of only two crash reports. In contrast, builds with one or more failing tests are associated with a median of 508 and 291 crash reports for Beta and Production builds, respectively. We further investigate the impact of ``flaky'' tests, which can both pass and fail on the same build, and find that they have a median of 514 and 234 crash reports for Beta and Production builds. Finally, building on previous research that has shown that tests that have failed frequently in the past will fail frequently in the future, we find that Builds with HighFailureTests have a median of 585 and 780 crash reports for Beta and Production builds. Unlike other types of test failures, HighFailureTests have a larger impact on Production releases than on Beta builds, and they have a median of 2.7 times more crashes than builds with normal test failures. We conclude that ignoring test failures is related to a dramatic increase in the number of crashes reported by users.
DOI 原文 ·
@inproceedings{paperbot4684,
title = {The impact of failing, flaky, and high failure tests on the number of crash reports associated with Firefox builds},
author = {Md Tajmilur Rahman and Peter C. Rigby},
booktitle = {FSE},
year = {2018},
doi = {10.1145/3236024.3275529}
}