一句话：提出全分片专家并行 FSEP，通过训练中专家重布局与通信调度缓解 MoE 负载不均，最高加速 1.69×。
专家并行是训练 MoE 模型的重要方式，但动态路由导致专家间负载严重不均衡，少数过载专家拖慢整体迭代。本文提出 LAER-MoE 训练框架，核心是全新并行范式 Fully Sharded Expert Parallel (FSEP)：按设备数完全分片每个专家参数，并在训练中通过 All-to-All 通信以专家粒度恢复部分专家，从而灵活重布局专家参数以增强负载均衡。为降低开销，作者对通信操作进行细粒度调度，并设计负载均衡规划器，在训练中制定专家重布局策略与 token 路由方案。在 A100 集群上的实验表明，相比当前最优训练系统最高取得 1.69 倍加速。源码已公开。
Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different input data. However, during expert parallel training, dynamic routing results in significant load imbalance among experts: a handful of overloaded experts hinder overall iteration, emerging as a training bottleneck. In this paper, we introduce LAER-MoE, an efficient MoE training framework. The core of LAER-MoE is a novel parallel paradigm, Fully Sharded Expert Parallel (FSEP), which fully partitions each expert parameter by the number of devices and restores partial experts at expert granularity through All-to-All communication during training. This allows for flexible re-layout of expert parameters during training to enhance load balancing. In particular, we perform fine-grained scheduling of communication operations to minimize communication overhead. Additionally, we develop a load balancing planner to formulate re-layout strategies of experts and routing schemes for tokens during training. We perform experiments on an A100 cluster, and the results indicate that our system achieves up to 1.69x acceleration compared to the current state-of-the-art training systems. Source code available at https://github.com/PKU-DAIR/Hetu-Galvatron/tree/laer-moe.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2602-11686,
title = {LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training},
author = {Xinyi Liu and Yujie Wang and Fangcheng Fu and Xuefeng Xiao and Huixia Li and Jiashi Li and Bin Cui},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790180}
}