一句话:提出 Khuzdul 分布式执行引擎,通过可扩展嵌入抽象与 BFS-DFS 混合探索,实现高效可扩展的分布式图模式挖掘。
现有分布式图模式挖掘(GPM)系统在划分图上性能与可扩展性难以兼顾,复制图方案则受限于内存与通信开销。本文提出 Khuzdul,一种具有明确抽象的分布式执行引擎,可与现有单机 GPM 系统集成。其核心是可扩展嵌入抽象,能表达模式枚举算法、支持细粒度任务调度,并实现低成本 GPM 专用数据复用以降低通信开销。BFS-DFS 混合探索生成充足并发任务,实现通信-计算重叠且内存消耗有界。作者将 Automine 和 GraphPi 移植到 Khuzdul 上构建两个分布式系统。评估表明,基于 Khuzdul 的系统显著优于最先进的划分图分布式 GPM 系统,最高加速 75.5 倍(平均 19.0 倍),与复制图最快系统性能相当或更优,并可在 commodity cluster 上扩展到超过一千亿条边的图。
This paper proposes Khuzdul, a distributed execution engine with a well-defined abstraction that can be integrated with existing single-machine graph pattern mining (GPM) systems to provide efficiency and scalability at the same time. The key novelty is the extendable embedding abstraction which can express pattern enumeration algorithms, allow fine-grained task scheduling, and enable low-cost GPM-specific data reuse to reduce communication cost. The effective BFS-DFS hybrid exploration generates sufficient concurrent tasks for communication-computation overlapping with bounded memory consumption. Two scalable distributed GPM systems are implemented by porting Automine and GraphPi on Khuzdul. Our evaluation shows that Khuzdul based systems significantly outperform state-of-the-art distributed GPM systems with partitioned graphs by up to 75.5× (on average 19.0×), achieve similar or even better performance compared with the fastest distributed GPM systems with replicated graph, and scale to massive graphs with more than one hundred billion edges with a commodity cluster.