一句话：通过支持已有工业测试用例播种与外部服务 Mock，将白盒模糊测试器 EvoMaster 落地到美团 40 个企业级 RPC API 并验证其增量收益。
微服务已成为工业界构建大规模 Web 服务的主流架构，但其高复杂度使得自动化系统级测试成为迫切需求。本文以开源白盒模糊测试器 EvoMaster（在已有实证研究中按行覆盖率和缺陷检测均表现最佳）为对象，开展工业落地实验：一方面扩展该工具以支持播种（seeding），即将已有工业测试用例纳入模糊测试的输入；另一方面支持对外部服务进行 Mock，并把 Mock 数据作为模糊测试输入的一部分统一处理。作者在美团 40 个基于 RPC 的企业级 API（核心业务逻辑共 560 万行代码）上，以两种配置和两种时间预算进行评估。结果显示，相较于现有的人工系统级测试以及在线流量录制回放所生成的测试，EvoMaster 带来明显的额外收益：最佳配置可达 32.4% 的行覆盖率，在 40 个案例中的 36 个（90%）上覆盖率超过 10%，平均每个 API 识别出 3520 个潜在缺陷。论文还进一步总结并讨论了企业级微服务模糊测试中未来必须解决的重要挑战。
Microservices is now becoming a promising architecture to build large-scale web services in industry. Due to the high complexity of enterprise microservices, industry has an urgent need to have a solution to enable automated testing of such systems. EvoMaster is an open-source fuzzer, equipped with the state-of-the-art techniques for supporting automated system-level testing of Web APIs. It has been assessed as the most performant tool in two recent empirical studies in terms of line coverage and fault detection. In this paper, we carried out an empirical experiment to investigate how to better apply the state-of-the-art academic prototype (i.e., EvoMaster) in industrial context. We extended the tool to handle seeding of existing industrial tests, and mocking of external services with their data handled as part of the input fuzzing. We studied two configurations of EvoMaster, using two time budgets, on 40 enterprise RPC-based APIs (involving 5.6 million lines of code for their core business logic) at Meituan. Results show that, compared to existing practice of manual system-level testing and tests produced by record and replay of online traffic, EvoMaster demonstrates clear additional benefits. EvoMaster with the best configuration is capable of covering up to 32.4% line coverage, covering more than 10% line coverage on 36 out of 40 (90%) case studies, and identifying on average 3520 potential faults in these 40 APIs. In addition, we also identified and discussed important challenges in fuzzing enterprise microservices that must be addressed in the future.CCS CONCEPTS• Software and its engineering → Search-based software engineering; Software verification and validation.
@inproceedings{0001ATXW24,
title = {Seeding and Mocking in White-Box Fuzzing Enterprise RPC APIs: An Industrial Case Study},
author = {Man Zhang and Andrea Arcuri and Piyun Teng and Kaiming Xue and Wenhao Wang},
booktitle = {ASE},
year = {2024},
doi = {10.1145/3691620.3695265}
}