一句话：提出基于Lyapunov稳定性控制理论的测试预言LPEA，用自动判定替代人类专家判定RL软件输出正确性。
强化学习软件日益复杂，其输出正确性难以定义，导致测试预言问题；现有方法依赖人类专家判定，难以自动化且受专家质量限制。本文利用Lyapunov稳定性控制理论设计RL软件测试预言：在训练中引入Lyapunov稳定性概念，假设正确实现的RL软件应输出满足该理论的智能体，据此提出LPEA(ϑ, θ)。作者在代表性RL算法和软件缺陷上实验，与人类预言对比。结果表明LPEA在多数指标上优于人类预言；例如LPEA(100%,75%)在准确率、精确率、召回率、F1和AUC上分别提升53.6%、50%、18.4%、34.8%和31.7%。
Reinforcement Learning (RL) has gained significant attention in recent years. As RL software becomes more complex and infiltrates critical application domains, ensuring its quality and correctness becomes increasingly important. An indispensable aspect of software quality/correctness insurance is testing. However, testing RL software faces unique challenges compared to testing traditional software, due to the difficulty on defining the outputs' correctness. This leads to the RL test oracle problem. Current approaches to testing RL software often rely on human oracles, i.e. convening human experts to judge the correctness of RL software outputs. This heavily depends on the availability and quality (including the experiences, subjective states, etc.) of the human experts, and cannot be fully automated. In this paper, we propose a novel approach to design test oracles for RL software by leveraging the Lyapunov stability control theory. By incorporating Lyapunov stability concepts to guide RL training, we hypothesize that a correctly implemented RL software shall output an agent that respects Lyapunov stability control theories. Based on this heuristics, we propose a Lyapunov stability control theory based oracle,$\operatorname{LPEA}(\vartheta, \theta)$, for testing RL software. We conduct extensive experiments over representative RL algorithms and RL software bugs to evaluate our proposed oracle. The results show that our proposed oracle can outperform the human oracle in most metrics. Particularly, LPEA$(\vartheta=100 \%, \theta=75 \%)$outperforms the human oracle by$53.6 \%, 50 \%, 18.4 \%, 34.8 \%, 18.4 \%, 127.8 \%, 60.5 \%, 38.9 \%$, and 31.7 % respectively on accuracy, precision, recall, F1 score, true positive rate, true negative rate, false positive rate, false negative rate, and ROC curve's AUC; and LPEA ($\vartheta=100 \%, \theta=50 \%$) outperforms the human oracle by$48.2 \%, 47.4 \%, 10.5 \%, 29.1 \%$,$10.5 \%, 127.8 \%, 60.5 \%, 22.2 \%$, and 26.0 % respectively on these metrics.
@inproceedings{paperbot8692,
title = {A Test Oracle for Reinforcement Learning Software Based on Lyapunov Stability Control Theory},
author = {Shiyu Zhang and Haoyang Song and Qixin Wang and Henghua Shen and Yu Pei},
booktitle = {ICSE},
year = {2025},
doi = {10.1109/icse55347.2025.00074}
}