一句话：提出Libro框架，利用大语言模型从通用缺陷报告中生成复现测试，并通过后处理与排序提升有效性。
现有自动测试生成技术多以提高覆盖率或生成探索性输入为目标，难以实现更语义化的目标，例如根据缺陷报告生成复现测试。然而，实证研究表明开源仓库中因issue新增的测试约占项目测试套件规模的28%，缺陷复现对开发者至关重要。由于将缺陷报告中的期望程序语义转化为测试预言十分困难，现有故障复现技术通常仅处理程序崩溃这一小部分缺陷。为从通用缺陷报告中自动生成测试，本文提出Libro框架，使用已被证明能完成代码相关任务的大语言模型（LLM）。由于LLM本身无法执行目标缺陷代码，工作重点放在后处理步骤，以判断LLM何时有效，并按有效性对生成的测试排序。在Defects4J基准上，Libro为750个案例中的251个（33%）生成了失败复现测试，并对149个缺陷将复现测试排在首位。为缓解数据污染，作者还在LLM训练数据截止后提交的31个缺陷报告上评估，Libro为32%的报告生成了复现测试。
Many automated test generation techniques have been developed to aid developers with writing tests. To facilitate full automation, most existing techniques aim to either increase coverage, or generate exploratory inputs. However, existing test generation techniques largely fall short of achieving more semantic objectives, such as generating tests to reproduce a given bug report. Reproducing bugs is nonetheless important, as our empirical study shows that the number of tests added in open source repositories due to issues was about 28% of the corresponding project test suite size. Meanwhile, due to the difficulties of transforming the expected program semantics in bug reports into test oracles, existing failure reproduction techniques tend to deal exclusively with program crashes, a small subset of all bug reports. To automate test generation from general bug reports, we propose Libro, a framework that uses Large Language Models (LLMs), which have been shown to be capable of performing code-related tasks. Since LLMs themselves cannot execute the target buggy code, we focus on post-processing steps that help us discern when LLMs are effective, and rank the produced tests according to their validity. Our evaluation of Libro shows that, on the widely studied Defects4J benchmark, Libro can generate failure reproducing test cases for 33% of all studied cases (251 out of 750), while suggesting a bug reproducing test in first place for 149 bugs. To mitigate data contamination (i.e., the possibility of the LLM simply remembering the test code either partially or in whole), we also evaluate Libro against 31 bug reports submitted after the collection of the LLM training data terminated: Libro produces bug reproducing tests for 32% of the studied bug reports. Overall, our results show Libro has the potential to significantly enhance developer efficiency by automatically generating tests from bug reports.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2209-11515,
title = {Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction},
author = {Sungmin Kang and Juyeon Yoon and Shin Yoo},
booktitle = {ICSE},
year = {2023},
doi = {10.1109/icse48619.2023.00194}
}