一句话：提出 TLT 系统，通过自适应推测解码无损加速推理强化学习训练，缓解长尾生成瓶颈。
大语言模型通过强化学习训练推理能力时，响应生成呈现持续的长尾分布：少数超长响应主导执行时间，浪费资源并推高成本。为此，论文提出 TLT 系统，将自适应推测解码引入 RL 训练以无损加速。其挑战在于动态工作负载、不断演化的目标模型以及 draft 模型训练开销。TLT 通过两个协同组件解决：自适应 Drafter 在长尾生成期间利用空闲 GPU 持续训练轻量 draft 模型，使其与目标模型保持对齐且不增加额外成本；自适应 Rollout 引擎维护内存高效的预捕获 CUDAGraph 池，并为每个输入批次自适应选择合适的推测解码策略。评估表明，TLT 相比最先进系统实现超过 1.7 倍的端到端 RL 训练加速，保持模型精度，并免费产出适合高效部署的高质量 draft 模型。
The emergence of Large Language Models (LLMs) with strong reasoning capabilities marks a significant milestone, unlocking new frontiers in complex problem-solving. However, training these reasoning models, typically using Reinforcement Learning (RL), encounters critical efficiency bottlenecks: response generation during RL training exhibits a persistent long-tail distribution, where a few very long responses dominate execution time, wasting resources and inflating costs. To address this, we propose TLT, a system that accelerates reasoning RL training losslessly by integrating adaptive speculative decoding. Applying speculative decoding in RL is challenging due to the dynamic workloads, evolving target model, and draft model training overhead. TLT overcomes these obstacles with two synergistic components: (1) Adaptive Drafter, a lightweight draft model trained continuously on idle GPUs during long-tail generation to maintain alignment with the target model at no extra cost; and (2) Adaptive Rollout Engine, which maintains a memory-efficient pool of pre-captured CUDAGraphs and adaptively select suitable SD strategies for each input batch. Evaluations demonstrate that TLT achieves over 1.7x end-to-end RL training speedup over state-of-the-art systems, preserves the model accuracy, and yields a high-quality draft model as a free byproduct suitable for efficient deployment. Code is released at https://github.com/mit-han-lab/fastrl.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{abs-2511-16665,
title = {Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Drafter},
author = {Qinghao Hu and Shang Yang and Junxian Guo and Xiaozhe Yao and Yujun Lin and Yuxian Gu and Han Cai and Chuang Gan and Ana Klimovic and Song Han},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790231}
}