一句话:构建首个长时间运行测试套件数据集 LRTS,系统重新评估 59 种测试用例优先级排序技术。
持续集成中长时间运行的测试套件会延迟开发者反馈,测试用例优先级排序(TCP)通过重排测试以尽早暴露故障,对这类套件尤为重要。然而已有 TCP 研究多基于十多年前、套件仅持续数分钟的过时构建,或依赖不可获取的专有项目。本文提出 LRTS,这是首个长时间运行测试套件数据集,包含来自 10 个使用 Jenkins 的大型开源项目的 21,255 个 CI 构建与 57,437 次测试套件运行,时间跨度为 2020 至 2023 年,平均单次套件运行达 6.5 小时。基于 LRTS,作者评估了 59 种主流 TCP 技术,分析了混杂性测试失败对 TCP 的影响,并研究了针对无历史失败记录的失败测试的优先级排序。研究复现了 9 条既有结论、反驳了 2 条,并建立了 3 条新结论,表明优先运行更快且最近失败的测试效果最佳,优于复杂技术。
The prolonged continuous integration (CI) runs are affecting timely feedback to software developers. Test-case prioritization (TCP) aims to expose faults sooner by reordering tests such that the ones more likely to fail are run earlier. TCP is thus especially important for long-running test suites. While many studies have explored TCP, they are based on outdated CI builds from over 10 years ago with test suites that last several minutes, or builds from inaccessible, proprietary projects. In this paper, we present LRTS, the first dataset of long-running test suites, with 21,255 CI builds and 57,437 test-suite runs from 10 large-scale, open-source projects that use Jenkins CI. LRTS spans from 2020 to 2023, with an average test-suite run duration of 6.5 hours. On LRTS, we study the effectiveness of 59 leading TCP techniques, the impact of confounding test failures on TCP, and TCP for failing tests with no prior failures. We revisit prior key findings (9 confirmed, 2 refuted) and establish 3 new findings. Our results show that prioritizing faster tests that recently failed performs the best, outperforming the sophisticated techniques.