一句话：构建首个从 GitHub 挖掘的 REST API 真实缺陷基准，含 607 个缺陷、分类体系与缺陷/修复版本，并揭示现有测试工具仅能检测 10% 的缺陷。
现有 REST API 测试与调试研究常任意选取 API 进行评估，缺乏对真实项目中可自动检测或修复缺陷的系统认识。本文提出 Defects4REST——首个公开的 REST API 项目缺陷基准，通过聚类、主题建模与人工验证的半自动化流程，从开源 GitHub 仓库中挖掘出 50 个项目的 607 个缺陷，并归纳为 6 类、13 个子类的分类体系。每个缺陷关联 issue 报告、开发者修改后的补丁文件与提交信息，并提供便捷的缺陷版本与修复版本访问及工件分析能力。作者用 11 个项目的 30 个缺陷样本评估了四款最先进的 REST API 测试工具，发现仅检测出 10%（3/30）的真实缺陷，原因包括语义验证、状态化测试、基础设施感知测试等九项根本性局限。基准与 issue 挖掘框架均已开源，以支持 REST API 测试与调试技术的可复现研究。
Existing research on REST API testing and debugging often relies on arbitrarily selected APIs for evaluation, leaving a critical gap in our understanding of real world defects in REST API based projects, particularly those that can be automatically detected or repaired. To address this gap, we present Defects4REST, the first public benchmark of defects in REST API projects, systematically mined from open source GitHub repositories. Using a semi-automated process that combines clustering, topic modeling, and manual validation, we build a dataset of 607 REST API defects from 50 projects and organize them into a taxonomy of 6 defect types and 13 sub-types. Each defect is linked to an issue report, the developer-modified versions of the patched files, and the corresponding commit messages. Defects4REST features easy access to the buggy and patched versions of projects for each defect and enables analysis of the associated artifacts. To evaluate current automated testing capabilities, we assess four state-of-the-art REST API testing tools on a sample of 30 defects from 11 projects. Our evaluation reveals that tools detect only 10% (3/30) of real-world defects due to nine fundamental limitations including semantic validation, stateful testing, and infrastructure-aware testing. We release both the benchmark and the issue mining framework to support reproducible research in REST APIs, enabling the development and evaluation of REST API testing and debugging techniques.
@inproceedings{paperbot11716,
title = {Defects4REST: A Benchmark of Real-World Defects to Enable Controlled Testing and Debugging Studies for REST APIs},
author = {Rahil Mehta and Pushpak Katkhede and Manish Motwani},
booktitle = {ICSE},
year = {2026},
doi = {10.1145/3744916.3787787}
}