一句话：利用磁盘多重挂载与 NVMe 持久预留做协议级 fencing 接管，无需额外副本即实现秒级有状态服务故障切换。
本文来自阿里云消息中间件研发运维团队，针对云上有状态服务灾备中的现实“三难”——低成本、高稳态性能与快速故障切换难以兼得——提出一套生产级秒级故障切换机制。设计上保持稳态数据路径不变，继续使用本地文件系统与云块存储；故障切换时借助磁盘 Multi-Attach 与 NVMe Persistent Reservation 实施协议级 fencing，使新节点无需等待 detach/attach 流程即可立即抢占，从而避免脑裂。作者还引入去中心化租约探测，以及运行在恢复关键路径上、遵循最小权限的节点侧 agent，负责抢占、挂载与切换编排。以 Apache RocketMQ 作为代表性负载的实验表明，在不预留额外物理副本、不引入额外写放大的前提下，端到端故障切换可在秒级完成，稳态吞吐与延迟接近原生单副本部署；系统已完成数万次灾备演练并投入生产运行。
We are the development and operations team of Alibaba Cloud's messaging middleware, running thousands of tenant-isolated, state-ful clusters in production. Through years of handling real incidents such as node crashes, I/O stalls, and network partitions, we observed a practical "trilemma" in stateful disaster recovery: low cost, high steady-state performance, and fast failover are hard to achieve at the same time, echoing well-known engineering trade-offs in distributed systems and Site Reliability Engineering (SRE) practice [7, 14, 16]. This paper presents a production-grade, second-level failover mechanism for cloud-deployed stateful services, aiming to balance the three under realistic engineering constraints. Our design keeps the steady-state data path unchanged by continuing to use the local file system and cloud block storage. During failover, we enforce protocol-level fencing to enable fast and safe takeover: leveraging disk Multi-Attach and NVMe Persistent Reservation, the new node can preempt immediately to prevent split-brain without waiting for the detach/attach workflow [4–6, 30]. We further introduce decentralized lease probing and a least-privilege, node-side agent that performs preemption, mounting, and switchover orchestration on the critical recovery path. Experiments using Apache RocketMQ as a representative workload show that, without reserving extra physical replicas and without introducing additional write amplification, our approach achieves end-to-end failover in seconds while keeping steady-state throughput and latency close to a native single-replica deployment. The system has completed tens of thousands of disaster-recovery drills and is now running in production.
@inproceedings{JinGJFLLCL26,
title = {Replication-Free Failover: Protocol-Fenced Takeover for Stateful Services},
author = {Rongtong Jin and Yinyou Gu and Juntao Ji and Yubao Fu and Tao Liu and Fuzhi Lai and Gaoyang Cai and Qingshan Lin},
booktitle = {FSE},
year = {2026},
doi = {10.1145/3803437.3805226}
}