一句话：提出 CoServe，利用专家依赖减少切换，在有限异构内存中高效服务 CoE 模型。
大型模型资源密集，而 CoE 集成多个专家模型可提升精度，但大量专家导致内存容量不足，跨内存与存储层频繁切换专家带来显著开销。CoServe 面向 CPU-GPU 异构、有限内存环境，利用 CoE 推理中的专家依赖特性，设计依赖感知请求调度器和依赖感知专家管理，减少不必要专家切换，并引入离线 profiler 自动搜索不同处理器与设备上的最优资源分配。在真实智能制造负载中，相比 SOTA 系统吞吐提升 4.5 至 12 倍。
Large language models like GPT-4 are resource-intensive, but recent advancements suggest that smaller, specialized experts can outperform the monolithic models on specific tasks. The Collaboration-of-Experts (CoE) approach integrates multiple expert models, improving the accuracy of generated results and offering great potential for precision-critical applications, such as automatic circuit board quality inspection. However, deploying CoE serving systems presents challenges to memory capacity due to the large number of experts required, which can lead to significant performance overhead from frequent expert switching across different memory and storage tiers. We propose CoServe, an efficient CoE model serving system on heterogeneous CPU and GPU with limited memory. CoServe reduces unnecessary expert switching by leveraging expert dependency, a key property of CoE inference. CoServe introduces a dependency-aware request scheduler and dependency-aware expert management for efficient inference. It also introduces an offline profiler to automatically find optimal resource allocation on various processors and devices. In real-world intelligent manufacturing workloads, CoServe achieves 4.5× to 12× higher throughput compared to state-of-the-art systems.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2503-02354,
title = {CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memory},
author = {Jiashun Suo and Xiaojian Liao and Limin Xiao and Li Ruan and Jinquan Wang and Xiao Su and Zhisheng Huo},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3676641.3715986}
}