一句话：提出 SAMOTA，用代理模型辅助多目标搜索生成测试数据，高效检测 DNN 使能系统在线测试中的安全违规。
在线测试是 DNN 使能系统（如自动驾驶系统 ADS）安全保障的关键环节：被测系统嵌入高保真仿真环境，以闭环方式与环境交互。然而自动生成能触发安全违规的多样测试数据面临三重困难：需同时满足多个安全需求、高保真仿真器运行代价极高、可能的测试输入空间过大而无法穷举。为此，作者提出 SAMOTA（Surrogate-Assisted Many-Objective Testing Approach），将现有多目标搜索式测试套件生成算法与模拟仿真器行为的代理模型（surrogate model）结合，用廉价的代理评估替代大量昂贵仿真，从而高效引导搜索。在由多个 DNN 组成的 ADS 系统 Pylot 与高保真驾驶仿真器 CARLA 上的实证评估表明，与当前最先进的多目标测试套件生成算法及随机搜索相比，SAMOTA 在发现未知安全需求违规方面显著更有效且更高效，被视为在线测试落地的关键使能技术。
With the recent advances of Deep Neural Networks (DNNs) in real-world applications, such as Automated Driving Systems (ADS) for self-driving cars, ensuring the reliability and safety of such DNN-enabled Systems emerges as a fundamental topic in software testing. One of the essential testing phases of such DNN-enabled systems is online testing, where the system under test is embedded into a specific and often simulated application environment (e.g., a driving environment) and tested in a closed-loop mode in interaction with the environment. However, despite the importance of online testing for detecting safety violations, automatically generating new and diverse test data that lead to safety violations presents the following challenges: (1) there can be many safety requirements to be considered at the same time, (2) running a high-fidelity simulator is often very computationally-intensive, and (3) the space of all possible test data that may trigger safety violations is too large to be exhaustively explored. In this paper, we address the challenges by proposing a novel approach, called SAMOTA (Surrogate-Assisted Many-Objective Testing Approach), extending existing many-objective search algorithms for test suite generation to efficiently utilize surrogate models that mimic the simulator, but are much less expensive to run. Empirical evaluation results on Pylot, an advanced ADS composed of multiple DNNs, using CARLA, a high-fidelity driving simulator, show that SAMOTA is significantly more effective and efficient at detecting unknown safety requirement violations than state-of-the-art many-objective test suite generation algorithms and random search. In other words, SAMOTA appears to be a key enabler technology for online testing in practice.
@inproceedings{Haq0B22,
title = {Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization},
author = {Fitash Ul Haq and Donghwan Shin and Lionel Briand},
booktitle = {ICSE},
year = {2022},
doi = {10.1145/3510003.3510188}
}