一句话:Morpheus:首个将偏序采样与冲突分析用于真实分布式系统的并发测试工具,在 Erlang 系统中发现 11 个新错误。
分布式系统虽广泛部署,却因并发操作与故障的意外交织而难以推理,可能引发未定义行为与严重后果。本文提出 Morpheus,这是首个将偏序采样(partial order sampling)——一种经形式化分析与实证验证、能对错误检测提供强概率保证的随机测试方法——应用于真实分布式系统的并发测试工具。Morpheus 进一步引入冲突分析,通过预测并聚焦于影响测试结果的操作来改进随机测试。该工具面向 Erlang 语言及框架。作者在四个流行的 Erlang 分布式系统上进行了评估,包括消息代理 RabbitMQ 与 Erlang 标准库中的分布式数据库 Mnesia。结果显示 Morpheus 十分有效:在每个被测系统中都发现了此前未知的错误,共计 11 个,全部是核心协议缺陷,可能导致死锁、意外崩溃或状态不一致。
Despite their wide deployment, distributed systems remain notoriously hard to reason about. Unexpected interleavings of concurrent operations and failures may lead to undefined behaviors and cause serious consequences. We present Morpheus, the first concurrency testing tool leveraging partial order sampling, a randomized testing method formally analyzed and empirically validated to provide strong probabilistic guarantees of error-detection, for real-world distributed systems. Morpheus introduces conflict analysis to further improve randomized testing by predicting and focusing on operations that affect the testing result. Inspired by the recent shift in building distributed systems using higher-level languages and frameworks, Morpheus targets Erlang. Evaluation on four popular distributed systems in Erlang including RabbitMQ, a message broker service, and Mnesia, a distributed database in the Erlang standard libraries, shows that Morpheus is effective: It found previously unknown errors in every system checked, 11 total, all of which are flaws in their core protocols that may cause deadlocks, unexpected crashes, or inconsistent states.