一句话：提出 HILOS 框架，将注意力计算卸载到近存储加速器，大幅提升长上下文 LLM 离线推理吞吐并降低能耗。
大语言模型生成式推理对计算与内存的需求给实际部署带来巨大挑战，基于卸载的批量推理通过将主机内存与存储纳入 GPU 内存层级来支持离线场景，但 KV cache 随批量大小与上下文长度增长，导致严重的 I/O 开销。本文提出 HILOS 框架，利用近存储处理（near-storage processing, NSP）提升离线推理吞吐。其核心是 attention near storage，将访存密集的注意力操作卸载到近存储加速器，减少系统互连上的数据传输。在此基础上，HILOS 引入三项优化：协同 X-cache 在卸载后利用主机空闲资源降低 KV cache 的 I/O；延迟 KV cache 回写隐藏存储写延迟并缓解写放大；内存高效的注意力加速器在 NSP 设备资源约束下维持长序列的高吞吐。作者在配备 16 块 SmartSSD 的真实系统上实现并评估，相比最先进的卸载式推理框架，吞吐最高提升 7.86 倍，能耗最多降低 85%。
The computational and memory demands of large language models for generative inference present significant challenges for practical deployment. One promising solution targeting offline inference is offloading-based batched inference, which extends the GPU's memory hierarchy with host memory and storage. However, it often suffers from substantial I/O overhead, primarily due to the large KV cache sizes that scale with batch size and context window length. In this paper, we introduce HILOS, a framework that boosts offline inference throughput using near-storage processing. The core of HILOS is attention near storage, which offloads memory-intensive attention operations to near-storage accelerators, reducing traffic across the system interconnect. Building on attention near storage, HILOS incorporates three additional optimizations. First, cooperative X-cache minimizes KV cache I/O by exploiting available host resources after offloading. Second, delayed KV cache writeback hides storage write latency and mitigates storage write amplification. Finally, a memory-efficient attention accelerator sustains high throughput for long sequences within the resource constraints of NSP devices. We implemented and evaluated HILOS on a real system equipped with 16 SmartSSDs. Compared to state-of-the-art offloading-based inference frameworks, HILOS achieves up to 7.86x throughput while reducing energy consumption by up to 85%. The source code for HILOS is available at https://github.com/hongsunjang/HILOS.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{JangSSN00026,
title = {A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs},
author = {Hongsun Jang and Jaeyong Song and Changmin Shin and Si Ung Noh and Jaewon Jung and Jisung Park and Jinho Lee},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790119}
}