一句话：提出 SwiftSpec，通过跨 GPU 解聚草稿/目标模型并融合 GEMM 与注意力内核，显著降低单请求 LLM 解码延迟。
SwiftSpec 面向交互式系统中大语言模型单请求解码的低延迟需求。现有推测解码将小草稿模型置于关键路径，且单请求小批量使跨 GPU 扩展受通信开销限制。SwiftSpec 将草稿与目标模型解聚到单节点同构 GPU，直接使用 NCCL 低延迟原语，并优化 GEMM 与注意力内核；实现约 3 千行自定义 CUDA 融合内核及演化树缓存，以维持 KV-cache 一致性并最大化两模型间复用。在 8×H800 单节点上，Llama-3-70B 达 347 tokens/s，比 8×H200 上 NVIDIA 基准快 1.3×；跨五个模型族和六个数据集平均解码加速 1.75×，480 条 Llama-3-70B 查询均更快，95 分位请求相比最佳开源基线加速 1.7×。
Low-latency, single-request decoding of large language models is critical for interactive systems with tight SLA demands. Prior work reduces latency through speculative decoding (combining a small draft model with a larger target model), but the draft model remains on the critical path, and communication overhead limits scaling across GPUs due to the small batch size associated with single-request decoding. To address these limitations, this paper introduces SwiftSpec: a system architecture that disaggregates draft and target models across homogeneous GPUs within a single node and utilizes NCCL-low-latency primitives directly to improve the performance of core GEMM and attention kernels. Our implementation includes 3k lines of custom CUDA for fused kernels and an evolving tree cache for KV-cache consistency and maximized reuse between draft and target models. On a single 8×H800 GPU node, SwiftSpec achieves 347 tokens/s for Llama-3-70B---1.3× faster than NVIDIA's own benchmarks on a higher-performance 8×H200 setup---and averages 1.75× faster decoding than state-of-the-art speculative decoding across five model families and six datasets. Specifically, we find that for Llama-3-70B SwiftSpec is significantly faster across all 480 tested queries, showing 1.7× speedup over the best open-source baseline for 95th percentile requests. Code for SwiftSpec will be available at https://github.com/ByteDance-Seed/SwiftSpec
@inproceedings{ZhangJJY0LLH26,
title = {SwiftSpec: Disaggregated Speculative Decoding and Fused Kernels for Low-Latency LLM Inference},
author = {Ziyi Zhang and Ziheng Jiang and Chengquan Jiang and Menghan Yu and Size Zheng and Haibin Lin and Xin Liu and Henry Hoffmann},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790246}
}