一句话：提出 CacheCoder，用检索作 L1 缓存、草稿模型作 L2 缓存的分层投机解码，加速 LLM 代码生成最高 2.64×。
大语言模型在代码生成上表现优异，但自回归解码推理效率低。投机解码通过草稿-验证机制加速：先高效生成多个草稿 token，再由 LLM 并行验证。现有方法分为基于模型的草稿（质量高，但草稿耗时随草稿模型推理步数增加）和基于检索的草稿（高效但质量通常较低）。本文先通过实证实验分析两类草稿的优劣，随后受计算机体系结构缓存机制启发提出 CacheCoder：让检索充当高速 L1 缓存以快速获取草稿，草稿模型充当能力更强的 L2 缓存以补偿 L1 未命中，从而在可能时利用复用的速度、在必要时依赖生成的精度。为结合这两个层次化过程，作者提出 Preliminary Prediction 与 Draft Expansion 两种定制策略，用草稿模型预过滤检索内容，并用草稿模型生成的 token 扩展草稿以形成更长的草稿。实验表明，CacheCoder 相比自回归解码最高实现 2.64× 加速，显著优于当前最先进方法。
While Large Language Models (LLMs) have demonstrated exceptional performance in code generation, their autoregressive decoding paradigm typically results in low inference efficiency. Speculative decoding has emerged as a solution to speed up inference through its draft–verification mechanism, where multiple draft tokens are generated efficiently and then verified in parallel by LLMs. Current speculative decoding approaches for code generation are broadly categorized into two types: model-based approaches that generate drafts using a lightweight draft model, and retrieval-based approaches that reuse code snippets from external corpora. In this work, we first conduct empirical experiments to analyze the strengths and weaknesses of retrieval-based and model-based drafting. We find that retrieval-based approach is efficient but usually low-quality, while model-based one is high-quality but the drafting time increases with inference steps of the draft model. Inspired by the cache mechanism in computer architecture, we propose a novel framework CacheCoder. In our design, retrieval acts as a high-speed Level-1 (L1) cache for rapid draft acquisition, while the draft model functions as a more capable Level-2 (L2) cache to compensate for L1 misses, enabling us to capitalize on the speed of reuse when possible and rely on the precision of generation when necessary. To combine these two hierarchical processes, we propose two customized strategies, i.e., Preliminary Prediction and Draft Expansion, which utilize the draft model to pre-filter retrieved contents and expand them with tokens generated by the draft model to form drafts with higher length. Experimental results show that CacheCoder achieves up to a 2.64× speedup compared to autoregressive decoding, significantly outperforming state-of-the-art approaches.
@inproceedings{paperbot12378,
title = {It Takes Two to Tango: Accelerating Code Generation via Cache-Inspired Hierarchical Speculative Decoding},
author = {Yinggang Qiu and Shangwen Wang and Yihao Qin and Mingyang Geng and Dezun Dong},
booktitle = {ASE},
year = {2026},
doi = {10.1145/3832783.3837454}
}