一句话：用训练截止后创建的无偏数据集实证表明，LLM 生成的测试预言变异得分接近人工预言。
测试预言的自动生成仍是未解难题：EvoSuite、Randoop 等主流测试用例生成器依赖隐式、基于规则或回归预言，难以捕获依赖被测程序语义的失败；形式规约虽可导出预言，但代价高昂。LLM 有望克服这些限制，然而现有少量研究使用的 Defects4J 等公开基准很可能已进入 LLM 训练数据，威胁结论有效性。本文开展 LLM 生成测试预言的实证研究，使用来自 135 个 Java 项目的 13,866 个在 LLM 训练截止日期之后创建的测试预言，构成无偏数据集。实验显示，LLM 生成的预言平均变异得分为 43%，与人工设计预言的 45% 相当；结果还表明，测试前缀与被测程序中调用的方法已提供足够信息以生成良好预言，而额外的代码上下文并未带来相关收益。这些发现为在自动测试中使用 LLM 提供了可操作洞见，并揭示其当前在生成复杂预言方面的局限。
Generation of thorough test oracles is an open problem. Popular test case generators, like EvoSuite and Randoop, rely on implicit, rule-based, and regression oracles that miss failures that depend on the semantics of the program under test. Formal specifications can yield test oracles but are expensive to create.Large Language Models (LLMs) have the potential to overcome these limitations. The few studies of using LLMs to generate test oracles use modest-sized public benchmarks, such as Defects4J, that are likely to be included in the LLM training data, which threatens the validity of the results.This paper presents an empirical study of the effectiveness of LLMs in generating test oracles. Our experiments use 13,866 test oracles, from 135 Java projects, that were created after the LLMs training cut-off dates. Thus, our dataset is unbiased.In our experiments, LLMs generated oracles with average mutation score of 43%—similar to the 45% score of human-designed test oracles. Our results also indicate that the test prefix and the methods called in the program under test provide sufficient information to generate good oracles, while additional code context does not bring relevant benefits. These findings provide actionable insights into using LLMs for automatic testing and highlight their current limitations in generating complex oracles.
DOI 原文 ·
@inproceedings{paperbot9371,
title = {Do LLMs Generate Useful Test Oracles? An Empirical Study with an Unbiased Dataset},
author = {Davide Molinelli and Luca Di Grazia and Alberto Martin-Lopez and Michael D. Ernst and Mauro Pezzè},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00031}
}