一句话:提出基于 Transformer 的统一神经方法 TOGA,从焦点方法上下文推断异常与断言测试预言,精度提升 33%,发现 57 个真实缺陷。
软件测试中,测试预言(通常为条件表达式)记录了单元在给定测试前缀下应有的行为,但自动合成功能测试预言十分困难,因为它必须刻画开发者意图的功能而非已实现的功能。本文提出 TOGA(Test Oracle GenerAtion),一种基于 Transformer 的统一神经方法,依据焦点方法的上下文同时推断异常型与断言型测试预言。该方法能够处理文档含糊或缺失、甚至缺少实现的单元。作者从预言推断准确率与功能缺陷发现两方面进行评估:在留出测试数据集上总体准确率达 96%,较现有预言推断方法提升 33%;将其与自动化测试生成工具 EvoSuite 集成后,在大型 Java 程序中发现了 57 个真实缺陷,其中 30 个在评估中未被任何其他自动化测试方法发现。
Testing is widely recognized as an important stage of the software development lifecycle. Effective software testing can provide benefits such as bug finding, preventing regressions, and documentation. In terms of documentation, unit tests express a unit's intended functionality, as conceived by the developer. A test oracle, typically expressed as an condition, documents the intended behavior of a unit under a given test prefix. Synthesizing a functional test oracle is a challenging problem, as it must capture the intended functionality rather than the implemented functionality. In this paper, we propose TOGA (a neural method for Test Oracle GenerAtion), a unified transformer-based neural approach to infer both exceptional and assertion test oracles based on the context of the focal method. Our approach can handle units with ambiguous or missing documentation, and even units with a missing implementation. We evaluate our approach on both oracle inference accuracy and functional bug-finding. Our technique improves accuracy by 33% over existing oracle inference approaches, achieving 96% over-all accuracy on a held out test dataset. Furthermore, we show that when integrated with a automated test generation tool (EvoSuite), our approach finds 57 real world bugs in large-scale Java programs, including 30 bugs that are not found by any other automated testing method in our evaluation.