一句话：提出图流水线并行 GPP，将 DNN 划分为依赖 DAG 的阶段，并实现 GraphPipe 系统以提升训练性能与扩展性。
DNN 规模增长使单设备训练不可行。现有流水线并行仅考虑顺序阶段，忽略 DNN 拓扑，错失模型并行机会。本文提出图流水线并行（GPP），将 DNN 划分为依赖关系由有向无环图描述的阶段；它推广顺序流水线并行，保留 DNN 固有拓扑，使计算无关算子并发执行，从而降低内存需求并提升 GPU 性能。作者实现分布式系统 GraphPipe，自动进行阶段图划分、微批次调度优化，并按 GPP 策略并行训练。在多种 DNN 上，GraphPipe 比 PipeDream 和 Piper 最高提速 1.6×，搜索时间减少 9–21×。
Deep neural networks (DNNs) continue to grow rapidly in size, making them infeasible to train on a single device (e.g. GPU). Pipeline parallelism is commonly used in existing DNN systems to support large-scale DNN training by partitioning a DNN into multiple stages, which concurrently perform DNN computation for different micro-batches of training samples in a pipeline fashion. However, existing pipeline-parallel approaches only consider sequential pipeline stages and thus ignore the topology of a DNN, resulting in missed model-parallel opportunities. This paper presents graph pipeline parallelism (GPP), a new pipeline-parallel scheme that partitions a DNN into pipeline stages whose dependencies are identified by a directed acyclic graph. GPP generalizes existing sequential pipeline parallelism and preserves the inherent topology of a DNN to enable concurrent execution of computationally-independent operators, resulting in reduced memory requirement and improved GPU performance. In addition, we develop GraphPipe, a distributed system that exploits GPP strategies to enable performant and scalable DNN training. GraphPipe partitions a DNN into a graph of stages, optimizes micro-batch schedules for these stages, and parallelizes DNN training using the discovered GPP strategies. Evaluation on a variety of DNNs shows that GraphPipe outperforms existing pipeline-parallel systems such as PipeDream and Piper by up to 1.6×. GraphPipe also reduces the search time by 9-21× compared to PipeDream and Piper.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2406-17145,
title = {<scp>GraphPipe:</scp>
Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelism},
author = {Byungsoo Jeon and Mengdi Wu and Shiyi Cao and Sunghyun Kim and Sunghyun Park and Neeraj Aggarwal and Colin Unger and Daiyaan Arfeen and Peiyuan Liao and Xupeng Miao and Mohammad Alizadeh and Gregory R. Ganger and Tianqi Chen and Zhihao Jia},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3669940.3707220}
}