一句话：提出 Aqua，通过时间片抢占调度与网络加速内存卸载降低 LLM 推理换页开销，兼顾高负载下的响应性与吞吐。
LLM 推理受限于 GPU 内存容量；云端 LLM 请求突增会耗尽 GPU 内存，使多个 prompt 争抢有限资源。现有服务引擎多用准入控制应对，导致突发期间无响应。本文主张对 prompt 进行时间片抢占调度，但抢占会带来高额 paging 开销并降低吞吐。Aqua 是一个 GPU 内存管理框架，通过网络加速的内存卸载显著降低推理状态换页开销，从而在突发请求下同时实现高响应性与高吞吐。作者在配备 8 张 Nvidia H100 80G GPU 的服务器上托管多种模态的先进大生成模型进行评估。结果显示，Aqua 相比 state-of-the-art 将 LLM 推理响应性提升 20 倍，并将单条长 prompt 下的推理吞吐提升 4 倍。
Inference on large-language models (LLMs) is constrained by GPU memory capacity. A sudden increase in the number of inference requests to a cloud-hosted LLM can deplete GPU memory, leading to contention between multiple prompts for limited resources. Modern LLM serving engines deal with the challenge of limited GPU memory using admission control, which causes them to be unresponsive during request bursts. We propose that preemptive scheduling of prompts in time slices is essential for ensuring responsive LLM inference, especially under conditions of high load and limited GPU memory. However, preempting prompt inference incurs a high paging overhead, which reduces inference throughput. We present Aqua, a GPU memory management framework that significantly reduces the overhead of paging inference state; achieving both responsive and high throughput inference even under bursty request patterns. We evaluate Aqua by hosting several state-of-the-art large generative ML models of different modalities on servers with 8 Nvidia H100 80G GPUs. Aqua improves the responsiveness of LLM inference by 20X compared to the state-of-the-art. It improves LLM inference throughput over a single long prompt by 4X.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{KumarAS25,
title = {A
<scp>qua</scp>
: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains},
author = {Abhishek Vijaya Kumar and Gianni Antichi and Rachee Singh},
booktitle = {ASPLOS},
year = {2025},
doi = {10.1145/3676641.3715983}
}