一句话：提出前缀感知的 decode 注意力内核 PAT，按共享前缀打包查询并复用 KV，平均降低注意力延迟 53.5%。
LLM 服务中解码阶段注意力日益成为主导开销，其受限于从全局内存反复加载海量 KV cache。真实负载普遍存在跨请求的层次化共享前缀（系统提示、工具/模板、RAG），但现有注意力实现未能充分利用：单 query 单 CTA 的执行方式会重复加载共享前缀 KV，而一刀切的 tiling 又使片上资源闲置，并在 KV 长度不均时加剧资源气泡，进而放大访存带宽压力。本文提出 PAT，一种面向 LLM 解码的前缀感知注意力内核，采用 pack-forward-merge 范式组织执行：先按共享前缀打包 query 以减少重复访存，再用定制的多 Tile 内核提升资源效率，并通过多流转发与 KV 切分减少资源气泡，最后以在线 softmax 完成合并，开销可忽略。PAT 作为即插即用插件集成到 vLLM。在真实与合成负载上，相较最先进的注意力内核，注意力延迟平均降低 53.5%，TPOT 降低 17.0–93.1%。
LLM serving is increasingly dominated by decode attention, which is a memory-bound operation due to massive KV cache loading from global memory. Meanwhile, real-world workloads exhibit substantial, hierarchical shared prefixes across requests (e.g., system prompts, tools/templates, RAG). Existing attention implementations fail to fully exploit prefix sharing: one-query-per-CTA execution repeatedly loads shared prefix KV cache, while one-size-fits-all tiling leaves on-chip resources idle and exacerbates bubbles for uneven KV lengths. These choices amplify memory bandwidth pressure and stall memory-bound decode attention. This paper introduces PAT, a prefix-aware attention kernel implementation for LLM decoding that organizes execution with a pack-forward-merge paradigm. PAT packs queries by shared prefix to reduce repeated memory accesses, runs a customized multi-tile kernel to achieve high resource efficiency. It further applies practical multi-stream forwarding and KV splitting to reduce resource bubbles. The final merge performs online softmax with negligible overhead. We implement PAT as an off-the-shelf plugin for vLLM. Evaluation on both real-world and synthetic workloads shows that PAT reduces attention latency by 53.5% on average and TPOT by 17.0-93.1% under the same configurations against state-of-the-art attention kernels. PAT's source code is publicly available at https://github.com/flashserve/PAT.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{YiZHYSWZZLL26,
title = {PAT: Accelerating LLM Decoding via
<u>P</u>
refix-
<u>A</u>
ware A
<u>t</u>
tention with Resource Efficient Multi-Tile Kernel},
author = {Jinjun Yi and Zhixin Zhao and Yitao Hu and Ke Yan and Weiwei Sun and Hao Wang and Laiping Zhao and Yuhao Zhang and Wenxin Li and Keqiu Li},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790200}
}