一句话：提出 Spindle 训练系统，用波前调度分解多任务多模态大模型执行，联合优化负载并行与依赖调度。
针对多任务多模态基础模型训练中复杂模型架构与异构任务/模态负载导致的系统效率挑战，本文提出 Spindle 训练系统。其核心思想是将模型执行分解为多个 wave，并将联合优化问题顺序求解，同时进行异构感知的工作负载并行化和依赖驱动的执行调度。该系统在多种多任务多模态模型上进行了实现与评估，实验表明其性能与效率优于现有最先进训练系统，最高可实现 71% 的加速。
Recent foundation models are capable of handling multiple tasks and multiple data modalities with the unified base model structure and several specialized model components. However, efficient training of such multi-task (MT) multi-modal (MM) models poses significant system challenges due to the sophisticated model architecture and the heterogeneous workloads of different tasks and modalities. In this paper, we propose Spindle, a brand new training system tailored for resource-efficient and high-performance training of MT MM models via wavefront scheduling. The key idea of Spindle is to decompose the model execution into waves and address the joint optimization problem sequentially, including both heterogeneity-aware workload parallelization and dependency-driven execution scheduling. We build our system and evaluate it on various MT MM models. Experiments demonstrate the superior performance and efficiency of Spindle, with speedup ratio up to 71% compared to state-of-the-art training systems.
@inproceedings{WangZFMZZHLC25,
title = {Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling},
author = {Yujie Wang and Shenhan Zhu and Fangcheng Fu and Xupeng Miao and Jie Zhang and Juan Zhu and Fan Hong and Yong Li and Bin Cui},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3676641.3715992}
}