paperbot · PL 论文追踪

全部论文 待读 收藏

P2 Revisiting Test-Case Prioritization on Long-Running Test Suites

重新审视长时间运行测试套件上的测试用例优先级排序
ISSTA Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis2024匹配度 2/10 · 该论文属于软件测试与持续集成中的测试用例优先级排序,构建了长时测试套件数据集并系统复现评估 59 种技术;与用户关注的静态分析、LLM/Agent、浮点程序分析及贝叶斯程序分析没有直接交集。仅在“重要场景下构建基准/工业落地”这一点上有间接参考价值,但研究任务本身不匹配。引用 4 inbox
Runxiang Cheng, Shuai Wang, Reyhaneh Jabbarvand, Darko Marinov
crossref: 2026-09-12

AI 速览

一句话:构建首个长时间运行测试套件数据集 LRTS,系统重新评估 59 种测试用例优先级排序技术。

持续集成中长时间运行的测试套件会延迟开发者反馈,测试用例优先级排序(TCP)通过重排测试以尽早暴露故障,对这类套件尤为重要。然而已有 TCP 研究多基于十多年前、套件仅持续数分钟的过时构建,或依赖不可获取的专有项目。本文提出 LRTS,这是首个长时间运行测试套件数据集,包含来自 10 个使用 Jenkins 的大型开源项目的 21,255 个 CI 构建与 57,437 次测试套件运行,时间跨度为 2020 至 2023 年,平均单次套件运行达 6.5 小时。基于 LRTS,作者评估了 59 种主流 TCP 技术,分析了混杂性测试失败对 TCP 的影响,并研究了针对无历史失败记录的失败测试的优先级排序。研究复现了 9 条既有结论、反驳了 2 条,并建立了 3 条新结论,表明优先运行更快且最近失败的测试效果最佳,优于复杂技术。

问题
现有测试用例优先级排序研究依赖十余年前的短时构建或不可获取的专有项目,缺乏对现代长时间运行测试套件的真实评估。
方法
构建 LRTS 数据集,涵盖 10 个大型开源 Jenkins 项目的 21,255 个 CI 构建和 57,437 次套件运行(2020–2023,平均 6.5 小时),并在其上系统评估 59 种 TCP 技术。
评估
通过复现与对比分析检验 9 条旧结论、反驳 2 条并新增 3 条结论,同时评估混杂性失败和无历史失败失败测试对 TCP 效果的影响。
原文摘要(Abstract)

The prolonged continuous integration (CI) runs are affecting timely feedback to software developers. Test-case prioritization (TCP) aims to expose faults sooner by reordering tests such that the ones more likely to fail are run earlier. TCP is thus especially important for long-running test suites. While many studies have explored TCP, they are based on outdated CI builds from over 10 years ago with test suites that last several minutes, or builds from inaccessible, proprietary projects. In this paper, we present LRTS, the first dataset of long-running test suites, with 21,255 CI builds and 57,437 test-suite runs from 10 large-scale, open-source projects that use Jenkins CI. LRTS spans from 2020 to 2023, with an average test-suite run duration of 6.5 hours. On LRTS, we study the effectiveness of 59 leading TCP techniques, the impact of confounding test failures on TCP, and TCP for failing tests with no prior failures. We revisit prior key findings (9 confirmed, 2 refuted) and establish 3 new findings. Our results show that prioritizing faster tests that recently failed performs the best, outperforming the sophisticated techniques.

链接与引用

DOI 原文 · PDF(开放获取) ·

BibTeX
@inproceedings{paperbot8507,
  title = {Revisiting Test-Case Prioritization on Long-Running Test Suites},
  author = {Runxiang Cheng and Shuai Wang and Reyhaneh Jabbarvand and Darko Marinov},
  booktitle = {ISSTA},
  year = {2024},
  doi = {10.1145/3650212.3680307}
}