一句话：提出 Concerto 编译器框架，将通信调度建模为资源受限项目调度问题并自动分解，实现大规模深度学习通信优化，无需手工调优。
深度学习规模指数增长带来可扩展性需求，而通信时间仍是训练瓶颈。现有优化与并行系统耦合、针对特定计算-通信重叠手工实现，性能、可编程性与通用性受限。本文提出 Concerto——自动优化与调度通信的编译器框架：将调度问题形式化为资源受限项目调度问题（RCPSP），用现成求解器求近最优调度，并通过自动分解为关键的同步通信创造重叠机会。评估显示 Concerto 可匹配或超越 Megatron-LM、JAX/XLA、DeepSpeed、Alpa 等含大量手工优化的先进框架；它将并行方案与通信优化解耦，无需人工优化即可泛化到多种并行策略。
With the exponential growth of deep learning (DL), there arises an escalating need for scalability. Despite significant advancements in communication hardware capabilities, the time consumed by communication remains a bottleneck during training. The existing various optimizations are coupled within parallel systems to implement specific computation-communication overlap. These approaches pose challenges in terms of performance, programmability, and generality. In this paper, we introduce Concerto, a compiler framework designed to address these challenges by automatically optimizing and scheduling communication. We formulate the scheduling problem as a resource-constrained project scheduling problem and use off-the-shelf solver to get the near-optimal scheduling. And use auto-decomposition to create overlap opportunity for critical (synchronous) communication. Our evaluation shows Concerto can match or outperform state-of-the-art parallel frameworks, including Megatron-LM, JAX/XLA, DeepSpeed, and Alpa, all of which include extensive hand-crafted optimization. Unlike previous works, Concerto decouples the parallel approach and communication optimization, then can generalize to a wide variety of parallelisms without manual optimization.
@inproceedings{ChengLDWWSLZD0025,
title = {Concerto: Automatic Communication Optimization and Scheduling for Large-Scale Deep Learning},
author = {Shenggan Cheng and Shengjie Lin and Lansong Diao and Hao Wu and Siyu Wang and Chang Si and Ziming Liu and Xuanlei Zhao and Jiangsu Du and Wei Lin and Yang You},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3669940.3707223}
}