一句话:利用 LLM 从大规模生产执行场景中识别未被测试覆盖的场景,并生成新测试用例以增强测试套件
测试套件天然不完善,可不断加入新用例以提升质量与可靠性。但在大型、长期维护的套件中,找到覆盖现有套件之外执行场景的用例极为困难且耗费人力。本文提出 E-Test,旨在缩小测试套件探索的执行空间与生产实际执行之间的差距。E-Test 利用大语言模型(LLM)从大规模执行场景(如生产监控数据)中识别尚未被测试的场景,并生成执行这些场景的新测试用例来增强套件。在 1,975 个执行场景的数据集上,E-Test 识别未测试场景的 F1 达 0.55,显著优于回归测试与现场测试方法(最高 0.34)及原生 LLM(最高 0.39),减少了套件维护的人工投入。
Test suites are inherently imperfect, and testers can always enrich a suite with new test cases that improve its quality and, consequently, the reliability of the target software system. However, finding test cases that explore execution scenarios beyond the scope of an existing suite can be extremely challenging and labor-intensive, particularly when managing large test suites over extended periods. In this paper, we propose E-Test, an approach that reduces the gap between the execution space explored with a test suite and the executions experienced in production. E-Test (i) identifies executions that have not yet been tested from large sets of scenarios, such as those monitored during intensive production usage, and (ii) generates new test cases that enhance the test suite. E-Test leverages Large Language Models (LLMs) to pinpoint scenarios that the current test suite does not adequately cover, and augments the suite with test cases that execute these scenarios. Our evaluation on a dataset of 1,975 execution scenarios demonstrates that E-Test identifies not-yet-tested execution scenarios significantly better than state-of-the-art approaches. While existing regression testing and field testing approaches for this task achieve a maximum F1-score of 0.34, and vanilla LLMs achieve a maximum F1-score of 0.39, E-Test reaches 0.55. These results highlight the impact of E-Test in enhancing test suites by effectively targeting not-yet-tested execution scenarios and reducing manual effort required for maintaining test suites.