一句话:回顾分布式深度学习算子内并行竞赛,发布基准并总结模型划分挑战与提交质量。
本文回顾了 ASPLOS 2025 / EuroSys 2025 分布式深度学习算子内并行竞赛。大规模深度学习依赖多加速器分布式计算,编译器需为模型操作选择合理并行策略;由于神经网络支持多种并行方式,为每条指令确定策略是困难的组合优化问题。为征集新思路,作者组织为期数月的竞赛,推动模型划分算法发展。文章介绍基本问题形式化、关键挑战与机会、新基准套件,并总结参赛提交质量与赛事经验,为后续自动并行与模型划分研究提供参考。
A chief enabler of large-scale deep learning is the distribution of computation across multiple interconnected hardware accelerators. In order to unlock the maximum possible performance, a compiler must first select a reasonable strategy to parallelize a model's operations. Since neural network architectures admit multiple flavors of parallelism, determining the proper strategy for each instruction is a critical (albeit non-trivial) task. To solicit new ideas toward solving this challenging combinatorial optimization problem, we organized the ASPLOS 2025 / EuroSys 2025 Contest on Intra-Operator Parallelism for Distributed Deep Learning, a multi-month competition focused on advancing the state-of-the-art for model partitioning algorithms. In this paper, we offer a retrospective of this event, including the basic problem formulation, key challenges & opportunities, our new benchmark suite, and the quality of submissions received.