一句话：提出 RhymeRL，利用相邻训练轮次 rollout 的相似性，以历史推测解码与两级调度加速 LLM 强化学习训练，最高提速 2.6 倍。
随着 LLM 发展，强化学习成为提升其推理能力的关键方法。RL 训练包含 rollout、reward 与 training 多阶段，需要多类 worker 协同，但现有系统 GPU 利用率低：rollout 阶段因 test-time scaling 占据主导，且同一批次内 rollout 长度不均衡造成 GPU 气泡；异步执行与截断虽能部分缓解，却可能牺牲训练精度。本文的洞察是相邻训练 epoch 的 rollout 响应高度相似。据此提出 RhymeRL：HistoSpec 利用历史 rollout token 序列获得准确草稿，实现推测解码推理引擎；HistoPipe 利用历史 rollout 分布的相似性，以两级调度均衡 rollout worker 的负载。实验表明，RhymeRL 在不损害精度、不改变 RL 范式的前提下，相比现有方法最高获得 2.6 倍性能提升。
With the rapid advancement of large language models (LLMs), reinforcement learning (RL) has emerged as a pivotal methodology for enhancing the reasoning capabilities of LLMs. Unlike traditional pre-training approaches, RL encompasses multiple stages: rollout, reward, and training, which necessitates collaboration among various worker types. However, current RL systems continue to grapple with substantial GPU underutilization, due to two primary factors: (1) The rollout stage dominates the overall RL process due to test-time scaling; (2) Imbalances in rollout lengths (within the same batch) result in GPU bubbles. While prior solutions like asynchronous execution and truncation offer partial relief, they may compromise training accuracy for efficiency. Our key insight stems from a previously overlooked observation: rollout responses exhibit remarkable similarity across adjacent training epochs. Based on the insight, we introduce RhymeRL, an LLM RL system designed to accelerate RL training with two key innovations. First, to enhance rollout generation, we present HistoSpec, a speculative decoding inference engine that utilizes the similarity of historical rollout token sequences to obtain accurate drafts. Second, to tackle rollout bubbles, we introduce HistoPipe, a two-tier scheduling strategy that leverages the similarity of historical rollout distributions to balance workload among rollout workers. Experimental results demonstrate that RhymeRL achieves up to a 2.6x performance improvement over existing methods, without compromising accuracy or modifying the RL paradigm.
@inproceedings{HeLF0LLX026,
title = {History Doesn't Repeat Itself but Rollouts Rhyme: Accelerating Reinforcement Learning with RhymeRL},
author = {Jingkai He and Tianjian Li and Erhu Feng and Dong Du and Qian Liu and Tao Liu and Yubin Xia and Haibo Chen},
booktitle = {ASPLOS},
year = {2026},
doi = {10.1145/3779212.3790172}
}