一句话：用大语言模型分析 OpenAPI 规范静态推断 REST API 的测试预言，F1 达 74.3%，发现 18 个真实缺陷。
REST API 测试用例生成工具发展迅速，但在测试预言（oracle）方面支持有限，通常只能检测崩溃、回归以及违反 API 规范或设计标准的情况。本文提出 SATORI（Static API Test ORacle Inference），一种黑盒方法：它分析 API 的 OpenAPI 规范，利用大语言模型根据响应字段的名称、描述等属性推断 API 的期望行为，从而自动生成测试预言。为便于落地，作者扩展了 PostmanAssertify，将 SATORI 生成的预言自动转换为可执行断言。在来自 12 个工业 API 的 17 个操作上的评估显示，SATORI 每个操作可生成多达数百个有效预言，F1 分数为 74.3%，在可比预言类型上超过需要实际执行 API 的动态方法 AGORA+（69.3%）。研究还表明静态与动态预言推断具有互补性，两者合计覆盖标注真值数据集中 90% 的预言。SATORI 在 Amadeus Hotel、Deutschebahn、FDIC、GitLab、Marvel、OMDb 与 Vimeo 等流行 API 中发现 18 个缺陷，并促成维护者更新文档。
REST API test case generation tools are evolving rapidly, with growing capabilities for the automated generation of complex tests. However, despite their strengths in test data generation, these tools are constrained by the types of test oracles they support, often limited to crashes, regressions, and noncompliance with API specifications or design standards. This paper introduces SATORI (Static API Test ORacle Inference), a black-box approach for generating test oracles for REST APIs by analyzing their OpenAPI Specification. SATORI uses large language models to infer the expected behavior of an API by analyzing the properties of the response fields of its operations, such as their name and descriptions. To foster its adoption, we extended the PostmanAssertify tool to automatically convert the test oracles reported by SATORI into executable assertions. Evaluation results on 17 operations from 12 industrial APIs show that SATORI can automatically generate up to hundreds of valid test oracles per operation. SATORI achieved an F1-score of 74.3%, outperforming the state-of-the-art dynamic approach AGORA+(69.3%)—which requires executing the API—when generating comparable oracle types. Moreover, our findings show that static and dynamic oracle inference methods are complementary: together, SATORI and AGORA+ found 90% of the oracles in our annotated ground-truth dataset. Notably, SATORI uncovered 18 bugs in popular APIs (Amadeus Hotel, Deutschebahn, FDIC, GitLab, Marvel, OMDb and Vimeo) leading to documentation updates by the API maintainers.
DOI 原文 ·
@inproceedings{paperbot9545,
title = {SATORI: Static Test Oracle Generation for REST APIs},
author = {Juan C. Alonso and Alberto Martin-Lopez and Sergio Segura and Gabriele Bavota and Antonio Ruiz-Cortés},
booktitle = {ASE},
year = {2025},
doi = {10.1109/ase63991.2025.00116}
}