一句话：提出DIP动态模态感知流水线调度，通过模态分段和细粒度子微批平衡负载，提升LMM训练吞吐。
大型多模态模型（LMM）可处理多模态输入，但训练效率受两个问题制约：异构模型架构导致流水线阶段不均衡，多模态数据多样性带来训练动态性。本文提出 DIP，一种动态、模态感知的流水线调度框架：将不同模态计算划分到专用流水线段，以在连续阶段内平衡负载；并动态把输入切分为更细粒度的模态特定子微批，以跨段平衡负载。DIP 利用空闲 CPU 异步生成调度，针对每个输入批次定制阶段执行而不阻塞训练。在 12B 到 94B 的五种 LMM（含视觉语言与扩散模型）上，吞吐相比最先进系统最高提升 97.3%。
Large multimodal models (LMMs) have demonstrated excellent capabilities in both understanding and generation tasks with various modalities. While these models can accept flexible combinations of input data, their training efficiency suffers from two major issues: pipeline stage imbalance caused by heterogeneous model architectures, and training data dynamicity stemming from the diversity of multimodal data. In this paper, we present DIP, a dynamic and modality-aware pipeline scheduling framework designed for LMM training. DIP tackles the challenge of dynamic imbalance via two key techniques: (1) separating computations of different modalities into dedicated pipeline segments to balance workloads within a continuous set of stages; (2) dynamically splitting input data into finer-grained, modality-specific sub-microbatches to balance workloads across these segments. By asynchronously generating pipeline schedules on idle CPU resources during training, DIP dynamically tailors stage executions to each input batch without stalling the training process. We validate DIP on a diverse set of five LMMs, ranging from 12B to 94B parameters and including vision-language and diffusion models. Experimental results show that our system achieves up to 97.3% higher throughput compared to state-of-the-art systems, demonstrating strong adaptability to fluctuating multimodal training workloads.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{XueH0JSM0JX026,
title = {DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipeline},
author = {Zhenliang Xue and Hanpeng Hu and Xing Chen and Yimin Jiang and Yixin Song and Zeyu Mi and Yibo Zhu and Daxin Jiang and Yubin Xia and Haibo Chen},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790154}
}