一句话：对神经测试预言生成方法 TOGA 开展大规模复现评估，揭示其误分类、低置信与假阳性等实际局限。
测试预言定义是测试开发的核心，但人工构建成本高昂。本文评估近期神经网络方法 TOGA：它基于 EvoSuite 生成的测试输入，产生异常与断言两类预言。作者在 Defects4j 上复现 TOGA 优于规约、搜索和神经网络技术的结果，并进一步在 25 个真实 Java 系统、22.35 万个测试用例、5.1 万个注入故障上开展外部、扩展与概念性复现。结果显示，TOGA 有 24% 的情况错误判断所需预言类型；正确分类时约 62% 因置信度不足而不生成断言；生成的断言中逾 47% 为假阳性，真阳性断言相对先前工作仅提升 0.3% 的故障检测。该工作揭示当前神经预言生成的局限，并为未来自动预言生成方法的评估提供经验。
Defining test oracles is crucial and central to test development, but manual construction of oracles is expensive. While recent neural-based automated test oracle generation techniques have shown promise, their real-world effectiveness remains a compelling question requiring further exploration and understanding. This paper investigates the effectiveness of TOGA, a recently developed neural-based method for automatic test oracle generation. TOGA utilizes EvoSuite-generated test inputs and generates both exception and assertion oracles. In a Defects4j study, TOGA outperformed specification, search, and neural-based techniques, detecting 57 bugs, including 30 unique bugs not detected by other methods. To gain a deeper understanding of its applicability in real-world settings, we conducted a series of external, extended, and conceptual replication studies of TOGA. In a large-scale study involving 25 real-world Java systems, 223.5K test cases, and 51K injected faults, we evaluate TOGA’s ability to improve fault-detection effectiveness relative to the state-of-the-practice and the state-of-the-art. We find that TOGA misclassifies the type of oracle needed 24% of the time and that when it classifies correctly around 62% of the time it is not confident enough to generate any assertion oracle. When it does generate an assertion oracle, more than 47% of them are false positives, and the true positive assertions only increase fault detection by 0.3% relative to prior work. These findings expose limitations of the state-of-the-art neural-based oracle generation technique, provide valuable insights for improvement, and offer lessons for evaluating future automated oracle generation methods.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{HossainFDEV23,
title = {Neural-Based Test Oracle Generation: A Large-Scale Evaluation and Lessons Learned},
author = {Soneya Binta Hossain and Antonio Filieri and Matthew B. Dwyer and Sebastian Elbaum and Willem Visser},
booktitle = {FSE},
year = {2023},
doi = {10.1145/3611643.3616265}
}