paperbot · PL 论文追踪

全部论文 待读 收藏

P2 SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification

SpecInfer：基于树的推测推理与验证加速大语言模型服务
ASPLOS Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 32024匹配度 2/10 · 论文聚焦于LLM服务加速的推测解码与树形并行验证，属于机器学习系统优化，未涉及静态分析、程序分析、浮点程序分析或贝叶斯程序分析等用户核心研究方向，与PL与AI交叉领域关联很弱。引用 445 inbox
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, Zhihao Jia
机器学习与程序语言
crossref: 2026-09-12dblp: 2026-09-12arxiv: 2026-09-12

AI 速览

一句话：提出 SpecInfer，用小型模型生成 token 树，并通过树形并行解码在 LLM 上验证，从而加速 LLM 服务并保持质量。

生成式大语言模型服务通常因逐 token 自回归解码而面临高延迟与高计算开销。SpecInfer 的核心思想是利用小型推测模型预测 LLM 的输出，并将预测组织为 token 树：树的每个节点表示一个候选 token 序列。系统使用一种新颖的树形并行解码机制，在 LLM 上并行验证 token 树中所有候选序列的正确性。与将 LLM 用作增量解码器不同，SpecInfer 将 LLM 用作 token 树验证器，从而显著降低端到端延迟与计算需求，并可证明地保持模型质量。评估表明，SpecInfer 在分布式 LLM 推理中比现有服务系统快 1.5–2.8 倍，在基于 offloading 的推理中快 2.6–3.5 倍，且生成性能不变。

问题
生成式 LLM 服务中自回归解码导致高延迟和高计算开销，需要在不损失生成质量的前提下加速推理。
方法
利用小型推测模型预测 LLM 输出并组织成 token 树，通过树形并行解码机制在 LLM 上并行验证所有候选序列，将 LLM 用作树验证器而非增量解码器。
评估
在分布式 LLM 推理和基于 offloading 的推理场景下与现有服务系统对比，分别取得 1.5–2.8 倍和 2.6–3.5 倍加速，同时保持相同生成性能。
原文摘要（Abstract）

This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality. Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1.5-2.8× for distributed LLM inference and by 2.6-3.5× for offloading-based LLM inference, while preserving the same generative performance. SpecInfer is publicly available at https://github.com/flexflow/FlexFlow/

链接与引用

DOI 原文 · arXiv · PDF（开放获取） · DBLP

BibTeX 
@inproceedings{MiaoOZCWZWZYSSC24,
  title = {SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification},
  author = {Xupeng Miao and Gabriele Oliaro and Zhihao Zhang and Xinhao Cheng and Zeyu Wang and Zhengxin Zhang and Rae Ying Yee Wong and Alan Zhu and Lijie Yang and Xiaoxiang Shi and Chunan Shi and Zhuoming Chen and Daiyaan Arfeen and Reyna Abhyankar and Zhihao Jia},
  booktitle = {ASPLOS},
  year = {2024},
  doi = {10.1145/3620666.3651335}
}