paperbot · PL 论文追踪

全部论文 待读 收藏

P2 MoE-L <scp>ightning</scp> : High-Throughput MoE Inference on Memory-constrained GPUs

MoE-Lightning：内存受限 GPU 上的高吞吐 MoE 推理
ASPLOS Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 12025匹配度 2/10 · 该论文属于机器学习系统/推理优化方向，聚焦MoE模型在内存受限GPU上的吞吐提升，未涉及静态分析、程序分析、浮点程序分析或贝叶斯程序分析等用户核心研究方向。虽然标签含“机器学习与程序语言”，但内容更偏系统与运行时优化，与用户画像中PL与AI的交叉点（如静态分析用于ML/Agentic程序、LLM改进静态分析）基本无关。引用 93 inbox
Shiyi Cao, Shu Liu, Tyler Griggs, Peter Schafhalter, Xiaoxuan Liu, Ying Sheng, Joseph E. Gonzalez, Matei Zaharia, Ion Stoica
机器学习与程序语言运行时与内存管理
crossref: 2026-09-12dblp: 2026-09-12arxiv: 2026-09-12

AI 速览

一句话：提出 MoE-Lightning 高吞吐 MoE 批量推理系统，通过 CGOPipe 与 HRM 性能模型在内存受限 GPU 上显著提升吞吐。

MoE 模型能以较低推理成本扩大模型容量，但参数量巨大，使其难以在内存受限或低端 GPU 上部署；现有卸载式推理系统吞吐低且资源利用率不足。本文提出高吞吐 MoE 批量推理系统 MoE-Lightning，引入 CPU-GPU-I/O 流水线调度 CGOPipe，并结合分页权重提高资源利用率；同时提出基于分层 Roofline 模型的性能模型 HRM，用于搜索更高吞吐的策略。在单张 T4（16GB）上，Mixtral 8x7B 的吞吐最高可达现有最优卸载推理系统的 10.3 倍；当吞吐受 GPU 内存限制时，可达到理论上界，且 CPU 内存减少 2-3 倍。系统还支持在 2-4 张 T4 等低成本 GPU 上对 Mixtral 8x22B、DBRX 等更大 MoE 进行高效批量推理。

问题
MoE 模型参数量大，难以在内存受限或低端 GPU 上高效部署；现有卸载式推理系统吞吐低且资源利用率不足。
方法
提出 MoE-Lightning，采用 CPU-GPU-I/O 流水线调度 CGOPipe 与分页权重，并设计基于 Hierarchical Roofline Model 的性能模型 HRM 来搜索高吞吐策略。
评估
在单张 T4（16GB）上对 Mixtral 8x7B 达到最高 10.3 倍于现有最优卸载推理系统的吞吐，并可达到 GPU 内存约束下的理论上界且 CPU 内存减少 2-3 倍；在 2-4 张 T4 上支持 Mixtral 8x22B 和 DBRX 等更大 MoE 的批量推理。
原文摘要（Abstract）

Efficient deployment of large language models, particularly Mixture of Experts (MoE) models, on resource-constrained platforms presents significant challenges in terms of computational efficiency and memory utilization. The MoE architecture, renowned for its ability to increase model capacity without a proportional increase in inference cost, greatly reduces the token generation latency compared with dense models. However, the large model size makes MoE models inaccessible to individuals without high-end GPUs. In this paper, we propose a high-throughput MoE batch inference system, MoE-Lightning, that significantly outperforms past work. MoE-Lightning introduces a novel CPU-GPU-I/O pipelining schedule, CGOPipe, with paged weights to achieve high resource utilization, and a performance model, HRM, based on a Hierarchical Roofline Model we introduce to help find policies with higher throughput than existing systems. MoE-Lightning can achieve up to (10.3x) higher throughput than state-of-the-art offloading-enabled LLM inference systems for Mixtral 8x7B on a single T4 GPU (16GB). When the theoretical system throughput is bounded by the GPU memory, MoE-Lightning can reach the throughput upper bound with 2-3x less CPU memory, significantly increasing resource utilization. MoE-Lightning also supports efficient batch inference for much larger MoEs (e.g., Mixtral 8x22B and DBRX) on multiple low-cost GPUs (e.g., 2--4 T4s).

链接与引用

DOI 原文 · arXiv · PDF（开放获取） · DBLP

BibTeX 
@inproceedings{CaoLGSL0GZS25,
  title = {MoE-L
                    <scp>ightning</scp>
                    : High-Throughput MoE Inference on Memory-constrained GPUs},
  author = {Shiyi Cao and Shu Liu and Tyler Griggs and Peter Schafhalter and Xiaoxuan Liu and Ying Sheng and Joseph E. Gonzalez and Matei Zaharia and Ion Stoica},
  booktitle = {ASPLOS},
  year = {2025},
  doi = {10.1145/3669940.3707267}
}