一句话：提出 Dr. DRL，将有意遗忘融入持续学习，通过优先适配关键技能来加速 DRL 系统在环境漂移下的自愈。
深度强化学习（DRL）已被用于 Netflix、Facebook 等大规模生产系统，但环境漂移会使其产生不良行为。持续学习（CL）是应对环境变化的自愈手段，然而当漂移幅度较大时，CL 会因灾难性遗忘、热启动失败和收敛缓慢而陷入漫长的甚至失败的自愈周期。本文提出 Dr. DRL，一种面向 DRL 系统的有效自愈方法：它在标准 CL 中引入新颖的“有意遗忘”机制，刻意抹除 DRL 系统的次要行为，从而系统性地优先适配关键的问题求解技能。作者使用成熟的 DRL 算法，在多种漂移环境中将 Dr. DRL 与 vanilla CL 进行对比。实验表明，Dr. DRL 平均将自愈时间与微调回合数分别减少 18.74% 和 17.72%，额外解决了 vanilla CL 未能解决的 19.63% 的漂移环境，并在两者均能解决的环境中将奖励保持甚至提升最多 45%。
Deep reinforcement learning (DRL) is increasingly applied in large-scale productions like Netflix and Facebook. As with most data-driven systems, DRL systems can exhibit undesirable behaviors due to environmental drifts, which often occur in constantly-changing production settings. Continual Learning (CL) is the inherent self-healing approach for adapting the DRL agent in response to the environment's conditions shifts. However, successive shifts of considerable magnitude may cause the production environment to drift from its original state. Recent studies have shown that these environmental drifts tend to drive CL into long, or even unsuccessful, healing cycles, which arise from inefficiencies such as catastrophic forgetting, warm-starting failure, and slow convergence. In this paper, we propose Dr. DRL, an effective self-healing approach for DRL systems that integrates a novel mechanism of intentional forgetting into vanilla CL (i.e., standard CL) to overcome its main issues. Dr. DRL deliberately erases the DRL system's minor behaviors to systematically prioritize the adaptation of the key problem-solving skills. Using well-established DRL algorithms, Dr. DRL is compared with vanilla CL on various drifted environments. Dr. DRL is able to reduce, on average, the healing time and fine-tuning episodes by, respectively, 18.74% and 17.72%. Dr. DRL successfully helps agents to adapt to 19.63% of drifted environments left unsolved by vanilla CL while maintaining and even enhancing by up to 45% the obtained rewards for drifted environments that are resolved by both approaches.
DOI 原文 · arXiv · PDF（开放获取） · DBLP
@inproceedings{YahmedBBK23,
title = {An Intentional Forgetting-Driven Self-Healing Method for Deep Reinforcement Learning Systems},
author = {Ahmed Haj Yahmed and Rached Bouchoucha and Houssem Ben Braiek and Foutse Khomh},
booktitle = {ASE},
year = {2023},
doi = {10.1109/ase56229.2023.00121}
}