一句话：提出 FSMoE，通过统一抽象、在线 profiling 与通信计算协同调度，加速稀疏 MoE 模型训练。
稀疏激活的混合专家（MoE）可降低大语言模型计算量，但其 token 路由、通信、专家计算与专家并行影响模型质量与训练效率，现有系统难以灵活适配并高效调度。FSMoE 提出灵活可扩展的训练系统：统一抽象并在线 profiling MoE 模块以支持跨实现调度；协同调度节点内与节点间通信和计算；设计自适应梯度划分和通信计算流水线调度。在最多 48 GPU 的集群上，FSMoE 支持四类常用路由函数，较现有实现最高加速 1.42×；在 1458 个 MoE 层上比 DeepSpeed-MoE 和 Tutel 快 1.18×–1.22×，在 GPT-2 和 Mixtral MoE 模型上快 1.19×–3.01×。
Recent large language models (LLMs) have tended to leverage sparsity to reduce computations, employing the sparsely activated mixture-of-experts (MoE) technique. MoE introduces four modules, including token routing, token communication, expert computation, and expert parallelism, that impact model quality and training efficiency. To enable ver- satile usage of MoE models, we introduce FSMoE, a flexible training system optimizing task scheduling with three novel techniques: 1) Unified abstraction and online profiling of MoE modules for task scheduling across various MoE implementations. 2) Co-scheduling intra-node and inter-node communications with computations to minimize communication overheads. 3) To support near-optimal task scheduling, we design an adaptive gradient partitioning method for gradient aggregation and a schedule to adaptively pipeline communications and computations. We conduct extensive experiments with configured MoE layers and real-world MoE models on two GPU clusters. Experimental results show that 1) our FSMoE supports four popular types of MoE routing functions and is more efficient than existing implementations (with up to a 1.42× speedup), and 2) FSMoE outperforms the state-of-the-art MoE training systems (DeepSpeed-MoE and Tutel) by 1.18×-1.22× on 1458 MoE layers and 1.19×-3.01× on real-world MoE models based on GPT-2 and Mixtral using a popular routing function. In this work, we present a flexible training system named FSMoE to optimize task scheduling. To achieve this goal: 1) we design unified abstraction and online profiling of MoE modules across various MoE implementations, 2) we co-schedule intra-node and inter-node communications with computations to minimize communication overhead, and 3) we design an adaptive gradient partitioning method for gradient aggregation and a schedule to adaptively pipeline communications and computations. Experimental results on two clusters up to 48 GPUs show that our FSMoE outperforms the state-of-the-art MoE training systems (DeepSpeed-MoE and Tutel) with speedups of 1.18x-1.22x on 1458 customized MoE layers and 1.19x-3.01x on real-world MoE models based on GPT-2 and Mixtral.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{PanLZSTWL025,
title = {FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models},
author = {Xinglin Pan and Wenxiang Lin and Lin Zhang and Shaohuai Shi and Zhenheng Tang and Rui Wang and Bo Li and Xiaowen Chu},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3669940.3707272}
}