一句话：提出通信划分空间与三层级调度框架 Centauri，提升大模型混合并行训练中通信与计算的重叠效率。
大语言模型训练依赖混合并行方法，需在分布式分区图中整合多种通信集合操作，通信瓶颈往往通过通信与计算重叠来缓解。然而现有重叠方法要么偏向细粒度算子融合，要么仅支持有限的操作调度，难以在异构训练环境中充分优化性能。本文提出 Centauri，一个包含全面通信划分与分层调度方案的框架。作者提出由三个固有抽象维度构成的划分空间：原语替换、拓扑感知的分组划分与负载划分，共同形成高效的优化空间；并将混合并行训练中的调度任务分解为算子、层、模型三个层级，以确定通信与计算算子的高效重叠方式。通过上述技术，Centauri 有效重叠通信延迟并提升硬件利用率。评估结果表明，在多种并行训练配置下，Centauri 相比主流方法最高取得 1.49 倍加速。
Efficiently training large language models (LLMs) necessitates the adoption of hybrid parallel methods, integrating multiple communications collectives within distributed partitioned graphs. Overcoming communication bottlenecks is crucial and is often achieved through communication and computation overlaps. However, existing overlap methodologies tend to lean towards either fine-grained kernel fusion or limited operation scheduling, constraining performance optimization in heterogeneous training environments. This paper introduces Centauri, an innovative framework that encompasses comprehensive communication partitioning and hierarchical scheduling schemes for optimized overlap. We propose a partition space comprising three inherent abstraction dimensions: primitive substitution, topology-aware group partitioning, and workload partitioning. These dimensions collectively create a comprehensive optimization space for efficient overlap. To determine the efficient overlap of communication and computation operators, we decompose the scheduling tasks in hybrid parallel training into three hierarchical tiers: operation, layer, and model. Through these techniques, Centauri effectively overlaps communication latency and enhances hardware utilization. Evaluation results demonstrate that Centauri achieves up to 1.49× speedup over prevalent methods across various parallel training configurations.
@inproceedings{ChenLZDSZY24,
title = {Centauri: Enabling Efficient Scheduling for Communication-Computation Overlap in Large Model Training via Communication Partitioning},
author = {Chang Chen and Xiuhong Li and Qianchao Zhu and Jiangfei Duan and Peng Sun and Xingcheng Zhang and Chao Yang},
booktitle = {ASPLOS},
year = {2024},
doi = {10.1145/3620666.3651379}
}