一句话：提出奖励引导的程序合成范式 ReGuS，用带循环、条件与过程调用的分层合成算法高效求解长时程机器人控制任务。
深度强化学习在机器人任务上取得了成功，但缺乏支持逻辑推理与泛化的归纳偏置，导致其在稀疏奖励、长时程的复杂控制任务中探索效率低下；面向强化学习的现有程序合成方法也因直接沿用强化学习搜索或仅模仿强化学习模型而继承了同样的局限。本文提出 ReGuS，一种奖励引导的合成范式，通过为循环设计可分解的搜索空间、按需合成条件语句以及为过程调用引入课程式合成，形成分层合成算法，从而有效压缩长时程、多阶段、过程化机器人控制任务的探索空间。在自动驾驶、运动控制与物体操作等挑战性机器人任务上的实验表明，ReGuS 显著优于最先进的强化学习算法与标准程序合成基线。
Deep reinforcement learning (RL) has led to encouraging successes in numerous challenging robotics applications. However, the lack of inductive biases to support logic deduction and generalization in the representation of a deep RL model causes it less effective in exploring complex long-horizon robot-control tasks with sparse reward signals. Existing program synthesis algorithms for RL problems inherit the same limitation, as they either adapt conventional RL algorithms to guide program search or synthesize robot-control programs to imitate an RL model. We propose ReGuS, a reward-guided synthesis paradigm, to unlock the potential of program synthesis to overcome the exploration challenges. We develop a novel hierarchical synthesis algorithm with decomposed search space for loops, on-demand synthesis of conditional statements, and curriculum synthesis for procedure calls, to effectively compress the exploration space for long-horizon, multi-stage, and procedural robot-control tasks that are difficult to address by conventional RL techniques. Experiment results demonstrate that ReGuS significantly outperforms state-of-the-art RL algorithms and standard program synthesis baselines on challenging robot tasks including autonomous driving, locomotion control, and object manipulation. CCS Concepts: • Software and its engineering → Automatic programming.
@article{CuiWQZ24,
title = {Reward-Guided Synthesis of Intelligent Agents with Control Structures},
author = {Guofeng Cui and Yuning Wang and Wenjie Qiu and He Zhu},
journal = {Proceedings of the ACM on Programming Languages},
volume = {8},
number = {PLDI},
year = {2024},
doi = {10.1145/3656447}
}