一句话：通过离线物化 CUDA 图和 KV 缓存初始化信息，降低无服务器 LLM 推理冷启动延迟。
无服务器模型推理面临严重冷启动延迟：需要将模型加载到设备，而 LLM 还额外引入 KV 缓存初始化阶段（分析并预留内存）和 CUDA 图捕获阶段（针对不同批大小动态构建图）。这些阶段对推理性能至关重要，却成为冷启动瓶颈。本文提出 Medusa，通过状态物化缓解长冷启动延迟：在离线阶段物化 CUDA 图及 KV 缓存初始化所需信息，在线阶段高效恢复；并引入离线-在线协同参数恢复与触发内核增强的内核地址恢复两项技术，处理 CUDA 图中的非确定性问题。Medusa 成功物化并恢复 10 个模型共 139364 个 CUDA 图节点，将模型加载延迟降低 42.5%；在真实 LLM 推理负载下，将首 token 时间（TTFT）尾延迟降低 53.0%。
Serverless is a promising paradigm to provide scalable, cost-efficient, and easy-to-use model inference services. However, the cold start of model inference functions requires loading models to the devices, which incurs high latencies and undermines the benefits of serverless computing. In LLMs, things get even worse since two extra stages are introduced: a KV cache initialization stage that profiles and anticipates memory reservation for KV cache, and a capturing stage which dynamically constructs CUDA graphs for different batch sizes. Both stages are paramount to the inference performance, but become the main culprit of cold start latency. This paper proposes Medusa to mitigate the long cold start latency through state materialization. Instead of dynamic profiling and construction in the runtime, Medusa materializes the CUDA graphs as well as the information needed by the KV cache initialization in the offline phase, and restores them efficiently in the online phase. Medusa further introduces two novel techniques -- offline-online cooperated parameters restoration and triggering-kernels enhanced kernel address restoration -- to tackle non-deterministic issues in CUDA graphs. Medusa successfully materializes and restores CUDA graphs across 10 models (with a total of 139364 CUDA graph nodes), and reduces the latency of model loading by 42.5%. Under real-world LLM inference workloads, Medusa reduces the tail latency of the time to first token (TTFT) by 53.0%.
@inproceedings{ZengXGCL25,
title = {<scp>Medusa:</scp>
Accelerating Serverless LLM Inference with Materialization},
author = {Shaoxun Zeng and Minhui Xie and Shiwei Gao and Youmin Chen and Youyou Lu},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3669940.3707285}
}