一句话：用搜索式模糊测试自动生成示范并搜索安全场景，无需专家知识地学习与修复安全高效的强化学习策略。
论文关注深度强化学习策略的可靠性与人力成本问题：传统「从示范中学习」（RLfD）依赖昂贵的人类专家示范。作者提出一个将基于搜索的测试（模糊测试）与 RLfD 相结合的框架。该框架先用搜索式模糊测试自动生成多样化的示范数据，替代人工示范用于 RLfD 训练；随后在黑盒环境中搜索安全攸关场景，以评估所得策略的安全性与鲁棒性；一旦检测到不安全行为，便再次通过模糊测试生成代表安全行为的示范，据此修复策略。实验表明，该框架在不需要任何专家知识的前提下，能够高效地学得高性能且安全的策略。
Reinforcement learning from demonstrations (RLfD) is a promising approach to improve the exploration efficiency of reinforcement learning (RL) by learning from expert demonstrations in addition to interactions with the environment. In this paper, we propose a framework that combines techniques from search-based testing with RLfD with the goal to raise the level of dependability of RL policies and to reduce human engineering effort. Within our framework, we provide methods for efficiently training, evaluating, and repairing RL policies. Instead of relying on the costly collection of demonstrations from (human) experts, we automatically compute a diverse set of demonstrations via search-based fuzzing methods and use the fuzz demonstrations for RLfD. To evaluate the safety and robustness of the trained RL agent, we search for safety-critical scenarios in the black-box environment. Finally, when unsafe behavior is detected, we compute demonstrations through fuzz testing that represent safe behavior and use them to repair the policy. Our experiments show that our framework is able to efficiently learn high-performing and safe policies without requiring any expert knowledge.
@inproceedings{paperbot11327,
title = {Learning and Repair of Deep Reinforcement Learning Policies from Fuzz-Testing Data},
author = {Martin Tappler and Andrea Pferscher and Bernhard K. Aichernig and Bettina Könighofer},
booktitle = {ICSE},
year = {2024},
doi = {10.1145/3597503.3623311}
}