一句话：提出 CodaMosa，在搜索式测试生成陷入覆盖率停滞时用 Codex 生成示例测试引导搜索，显著提升覆盖率。
搜索式软件测试（SBST）通过测试用例生成与变异为被测程序生成高覆盖率测试，但其效果依赖于能触发程序核心逻辑的测试用例；当搜索陷入覆盖率平台期时便难以继续探索。本文提出 CodaMosa，先运行 SBST 直至覆盖率改进停滞，再让预训练代码大模型 Codex 为覆盖不足的函数提供示例测试用例，从而将搜索重定向到更有用的搜索空间区域。作者在 486 个基准上评估，与纯 SBST 和纯 LLM 基线相比，CodaMosa 在显著更多基准上提升了覆盖率（分别为 173 和 279 个），而覆盖率下降的基准仅分别为 10 和 4 个。
Search-based software testing (SBST) generates high-coverage test cases for programs under test with a combination of test case generation and mutation. SBST's performance relies on there being a reasonable probability of generating test cases that exercise the core logic of the program under test. Given such test cases, SBST can then explore the space around them to exercise various parts of the program. This paper explores whether Large Language Models (LLMs) of code, such as OpenAI's Codex, can be used to help SBST's exploration. Our proposed algorithm, CodaMosa, conducts SBST until its coverage improvements stall, then asks Codex to provide example test cases for under-covered functions. These examples help SBST redirect its search to more useful areas of the search space. On an evaluation over 486 benchmarks, CodaMosa achieves statistically significantly higher coverage on many more benchmarks (173 and 279) than it reduces coverage on (10 and 4), compared to SBST and LLM-only baselines.
@inproceedings{LemieuxILS23,
title = {CodaMosa: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models},
author = {Caroline Lemieux and Jeevana Priya Inala and Shuvendu K. Lahiri and Siddhartha Sen},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00085}
}