一句话:以真实数据突变并保持真实性不变式,自动构造 oracle,为 LiDAR 感知系统高效生成真实多样的测试用例。
自主系统依赖感知组件理解环境,而感知误判曾导致严重的系统级失败。现有感知软件测试方法存在两方面不足:难以高效生成能反映真实世界性能的测试数据,且多样性不足以覆盖稀有但安全攸关的长尾场景;这对提供 3D 扫描、可在各种光照下工作的 LiDAR 感知系统尤为突出。本文提出一种新的 LiDAR 感知系统测试方法:以现有真实世界数据为基础,通过保持真实性不变式的突变生成真实且多样的测试用例,从而产生数据集中罕见的输入,并自动构造 oracle 以识别感知性能中潜在的安全攸关问题。作者实现了该方法并为五个先进 LiDAR 感知系统生成了超过 50,000 个测试输入,实验表明它能高效生成可触发实际部署后会产生真实后果的感知错误的用例,且误报率较低。
Autonomous systems rely on a perception component to interpret their surroundings, and when misinterpretations occur, they can and have led to serious and fatal system-level failures. Yet, existing methods for testing perception software remain limited in both their capacity to efficiently generate test data that translates to real-world performance and in their diversity to capture the long tail of rare but safety-critical scenarios. These limitations are particularly evident for perception systems based on LiDAR sensors, which have emerged as a crucial component in modern autonomous systems due to their ability to provide a 3D scan of the world and operate in all lighting conditions. To address these limitations, we introduce a novel approach for testing LiDAR-based perception systems by leveraging existing real-world data as a basis to generate realistic and diverse test cases through mutations that preserve realism invariants while generating inputs rarely found in existing data sets, and automatically crafting oracles that identify potentially safety-critical issues in perception performance. We implemented our approach to assess its ability to identify perception failures, generating over 50,000 test inputs for five state-of-the-art LiDAR-based perception systems. We found that it efficiently generated test cases that yield errors in perception that could result in real consequences if these systems were deployed and does so at a low rate of false positives.