一句话:通过存储与计算分离、机架内存储池化和 compaction 远程卸载,解决分布式 LSM 数据库负载不均与后台任务干扰。
分布式 LSM 数据库常因实例间负载不均以及 flush、compaction 和数据迁移等后台任务的干扰,出现吞吐下降和延迟升高。Hailstorm 将数据库存储引擎部署在分布式文件系统上,把存储与处理解耦,实现存储池化与 compaction 卸载。它池化机架内的存储设备,使每个存储引擎都能利用整机架的聚合容量与带宽,从而在不重新分片的情况下应对负载不均;同时将 compaction 任务卸载到远程节点,分散其影响。评估表明,在倾斜负载的 MongoDB 部署中,Hailstorm 实现负载均衡,平均吞吐提升 60%,尾延迟降低至多 5 倍;范围查询吞吐提升至多 22 倍;OLTP 工作负载成本节省 47-56%。
Distributed LSM-based databases face throughput and latency issues due to load imbalance across instances and interference from background tasks such as flushing, compaction, and data migration. Hailstorm addresses these problems by deploying the database storage engines over a distributed filesystem that disaggregates storage from processing, enabling storage pooling and compaction offloading. Hailstorm pools storage devices within a rack, allowing each storage engine to fully utilize the aggregate rack storage capacity and bandwidth. Storage pooling successfully handles load imbalance without the need for resharding. Hailstorm offloads compaction tasks to remote nodes, distributing their impact, and improving overall system throughput and response time. We show that Hailstorm achieves load balance in many MongoDB deployments with skewed workloads, improving the average throughput by 60%, while decreasing tail latency by as much as 5X. In workloads with range queries, Hailstorm provides up to 22X throughput improvements. Hailstorm also enables cost savings of 47-56% in OLTP workloads.