一句话：提出 RankF，用大语言模型与历史执行信息对测试排序，更快定位顺序依赖 flaky 测试的首个相关测试。
顺序依赖（OD）flaky 测试是否通过取决于测试运行顺序，开发者需找到决定其通过或失败的 OD 相关测试。已有工作通过在 OD 测试前反复运行不同测试来寻找，但未考虑各测试成为 OD 相关测试的可能性。本文提出 RankF：按成为 OD 相关测试的可能性对测试排序，以更快找到首个 OD 相关测试。RankF_L 训练大语言模型分析测试代码；RankF_O 分析先前的测试顺序执行信息。作者在 24 个开源项目的 155 个 OD 测试上评估，与已有基线相比，RankF 更快找到首个 OD 相关测试；视 OD 相关测试类型，RankF 中位耗时 9.4 至 14.1 秒，基线为 34.2 至 118.5 秒。
One major challenge of regression testing are flaky tests, i.e., tests that may pass in one run but fail in another run for the same version of code. One prominent category of flaky tests is order-dependent (OD) flaky tests, which can pass or fail depending on the order in which the tests are run. To help developers debug and fix OD tests, prior work attempts to automatically find OD-relevant tests, which are tests that determine whether an OD test passes or fails, depending on whether the OD-relevant tests run before or after the OD test. Prior work found OD-relevant tests by running different tests before the OD test, without considering each test's likelihood of being OD-relevant tests. We propose RankF to rank tests in order of likelihood of being OD-relevant tests, finding the first OD-relevant test for a given OD test more quickly. We propose two ranking approaches, each requiring different information. Our first approach,$\boldsymbol{RankF}_L$, relies on training a large-language model to analyze test code. Our second approach,$\boldsymbol{RankF}_O$, relies on analyzing prior test-order execution information. We evaluate our approaches on 155 OD tests across 24 open-source projects. We compare RankF against baselines from prior work, where we find that RankF finds the first OD-relevant test for an OD test faster than the best baseline; depending on the type of OD-relevant test, RankF takes 9.4 to 14.1 seconds on median, compared to the baseline's 34.2 to 118.5 seconds on median.
DOI 原文 ·
@inproceedings{paperbot8710,
title = {Ranking Relevant Tests for Order-Dependent Flaky Tests},
author = {Shanto Rahman and Bala Naren Chanumolu and Suzzana Rafi and August Shi and Wing Lam},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00178}
}