一句话：用低精度专家变体替换非关键缓存缺失专家，并结合动态加载、预取与缓存策略加速边缘设备上的 MoE 推理。
MoE 架构以较低的计算开销支撑大规模 LLM 的扩展，但其庞大的内存需求使边缘设备部署受限；传统专家卸载虽能缓解内存压力，却显著抬高推理延迟。本文提出 MoE-APEX——自适应精度（Adaptive Precision）专家卸载系统，核心思想是用低精度专家变体替换较不关键的缓存缺失专家，在降低专家加载延迟的同时保持模型精度。围绕 MoE 计算的自然层次，系统提出三项技术：token 级动态专家加载、层间自适应专家预取，以及序列级代价感知专家缓存策略，从而充分发挥混合精度专家推理的收益。系统在 Llama.cpp 之上实现，在多种边缘设备上相较当前最优 MoE 卸载系统取得 1.34x 至 9.75x 的解码加速，为资源受限环境下高效部署 MoE 提供了可行方案。
Mixture-of-experts (MoE) architectures enable scalable Large Language Models (LLMs) with reduced computational overhead, yet their deployment on memory-constrained edge devices is hindered by substantial memory demands. Traditional expert-offloading techniques mitigate memory constraints but often significantly increase inference latency. We introduce MoE-APEX, an Adaptive Precision EXpert offloading system that optimizes MoE inference for edge architectures by dynamically managing expert precision. Our core innovation is to replace less critical cache-miss experts with low-precision variants, reducing loading latency while maintaining accuracy. MoE-APEX introduces three innovative techniques that map the natural hierarchy of MoE computation: (1) a token-level dynamic expert loading mechanism, (2) a layer-level adaptive expert prefetching technique, and (3) a sequence-level cost-aware expert caching policy. These innovations enable MoE-APEX to leverage the benefits of mixed-precision expert inference fully. Implemented atop Llama.cpp, MoE-APEX achieves decoding speedups ranging from 1.34x to 9.75x compared to state-of-the-art MoE offloading systems across diverse edge devices, offering a robust solution for efficient MoE deployment in resource-constrained environments.
@inproceedings{Tang0HPWH0G26,
title = {MoE-APEX: An Efficient MoE Inference System with Adaptive Precision Expert Offloading},
author = {Peng Tang and Jiacheng Liu and Xiaofeng Hou and Yifei Pu and Jing Wang and Pheng-Ann Heng and Chao Li and Minyi Guo},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790187}
}